Few-shot learning (FSL) commonly requires a model to identify images (queries) that belong to classes unseen during training, based on a few labelled samples of the new classes (support set) as reference. So far, plenty of algorithms involve training data augmentation to improve the generalization capability of FSL models, but outlier queries or support images during inference can still pose great generalization challenges. In this work, to reduce the bias caused by the outlier samples, we generate additional test-class samples by combining original samples with suitable train-class samples via a generative image combiner. Then, we obtain averaged features via an augmentor, which leads to more typical representations through the averaging. We experimentally and theoretically demonstrate the effectiveness of our method, obtaining a test accuracy improvement proportion of around 10% (e.g., from 46.86% to 53.28%) for trained FSL models. Importantly, given a pretrained image combiner, our method is training-free for off-the-shelf FSL models, whose performance can be improved without extra datasets nor further training of the models themselves.

We present a novel extension to the family of Soft Actor-Critic (SAC) algorithms. We argue that based on the Maximum Entropy Principle, discrete SAC can be further improved via additional statistical constraints derived from a surrogate critic policy. Furthermore, by perturbing these constraints we demonstrate that an added robustness can be provided against potentially unseen or adversarial environments, which are essential for safe deployment of reinforcement learning agents in the real-world. We provide theoretical analysis and show empirical results on low data regimes for both in-distribution and out-of-distribution variants of Atari 2600 games.

We present a new loss function that addresses the out-of-distribution (OOD) network calibration problem. While many objective functions have been proposed to effectively calibrate models in-distribution, our findings show that they do not always fare well OOD. Based on the Principle of Maximum Entropy, we incorporate helpful statistical constraints observed during training, delivering better model calibration without sacrificing accuracy. We provide theoretical analysis and show empirically that our method works well in practice, achieving state-of-the-art calibration on both synthetic and real-world benchmarks. Our code is available at https://github.com/dexterdley/MaxEnt-Loss.

Few-shot learning (FSL) usually trains models on data from one set of classes, but tests them on data from a different set of classes, providing a few labeled support samples of the unseen classes as a reference for the trained model. Due to the lack of target-relevant training data, there is usually high generalization error with respect to the test classes. In this work, we conduct empirical explorations and propose an ensemble method (namely QuickBoost), which is efficient and effective for improving the generalization of FSL. Specifically, QuickBoost includes an alternative-architecture pretrained encoder with a one-vs-all binary classifier (namely FSL-Forest) based on random forest algorithm, and is ensembled with the off-the-shelf FSL models via logit-level averaging. Experiments on three benchmarks demonstrate that our method achieves state-of-the-art performance with good efficiency. Codes are available at https://github.com/WendyBaiYunwei/FSL-QuickBoost.

The existing image captioning approaches typically train a one-stage sentence decoder, which is difficult to generate rich fine-grained descriptions. On the other hand, multi-stage image caption model is hard to train due to the vanishing gradient problem. In this paper, we propose a coarse-to-fine multi-stage prediction framework for image captioning, composed of multiple decoders each of which operates on the output of the previous stage, producing increasingly refined image descriptions. Our proposed learning approach addresses the difficulty of vanishing gradients during training by providing a learning objective function that enforces intermediate supervisions. Particularly, we optimize our model with a reinforcement learning approach which utilizes the output of each intermediate decoder's test-time inference algorithm as well as the output of its preceding decoder to normalize the rewards, which simultaneously solves the well-known exposure bias problem and the loss-evaluation mismatch problem. We extensively evaluate the proposed approach on MSCOCO and show that our approach can achieve the state-of-the-art performance.

Language models based on recurrent neural networks have dominated recent image caption generation tasks. In this paper, we introduce a language CNN model which is suitable for statistical language modeling tasks and shows competitive performance in image captioning. In contrast to previous models which predict next word based on one previous word and hidden state, our language CNN is fed with all the previous words and can model the long-range dependencies in history words, which are critical for image captioning. The effectiveness of our approach is validated on two datasets: Flickr30K and MS COCO. Our extensive experimental results show that our method outperforms the vanilla recurrent neural network based language models and is competitive with the state-of-the-art methods.

We present a new algorithm for multi-region segmentation of 2D images with objects that may partially occlude each other. Our algorithm is based on the observation that human performance on this task is based both on prior knowledge about plausible shapes and taking into account the presence of occluding objects whose shape is already known - once an occluded region is identified, the shape prior can be used to guess the shape of the missing part. We capture the former aspect using a deep learning model of shape, for the latter, we simultaneously minimize the energy of all regions and consider only unoccluded pixels for data agreement. Existing algorithms incorporating object shape priors consider every object separately in turn and can't distinguish genuine deviation from the expected shape from parts missing due to occlusion. We show that our method significantly improves on the performance of a representative algorithm, as evaluated on both preprocessed natural and synthetic images. Furthermore, on the synthetic images, we recover the ground truth segmentation with good accuracy.

Which parts of an image evoke emotions in an observer? To answer this question, we introduce a novel problem in computer vision - predicting an Emotion Stimuli Map (ESM), which describes pixel-wise contribution to evoked emotions. Building a new image database, EmotionROI, as a benchmark for predicting the ESM, we find that the regions selected by saliency and objectness detection do not correctly predict the image regions which evoke emotion. Although objects represent important regions for evoking emotion, parts of the background are also important. Based on this fact, we propose using fully convolutional networks for predicting the ESM. Both qualitative and quantitative experimental results confirm that our method can predict the regions which evoke emotion better than both saliency and objectness detection.

Previous work has correctly identified that many of the tags that users provide on images are not visually relevant, and subsequently assume such user generated tags are noise or irrelevant. Another assumption about user generated tags is that the order of these tags provides no useful information for the prediction of tags on future images. We challenge the aforementioned assumptions, and provide a machine learning approach to the problem of personalized image tagging with the following contributions: 1.) We reformulate the personalized image tagging problem as a search/retrieval ranking problem, 2.) We leverage the order of tags, which does not always reflect visual relevance, provided by the user in the past as a cue to their tag preferences, similar to click data, 3.) We propose a technique to augment sparse user tag data (semi-supervision), and 4.) We demonstrate the efficacy of our method on a subset of Flickr images, showing improvement over our previous methods.

Most works using convolutional neural networks (CNN) show the efficacy of their methods in standard object recognition tasks, but not in abstract tasks such as emotion classification and memorability prediction, which are a subject of increasing importance (especially as machines become more autonomous, there is a need for semantic understanding). To verify whether CNN-based methods are effective in abstract tasks, we select 8 different abstract tasks in computer vision, evaluating the performance of 5 different CNN-based training approaches in these tasks. We show that CNN-based approaches outperform the state-of-the-art results in all the 8 tasks. Furthermore, we show that concatenating CNN features learned from different tasks can enhance the performance in each task. We also show that concatenating the CNN features learned from all the tasks under experiment does not perform the best, which is different from what is usually shown in previous works. Using CNN as a tool to correlate different tasks, we suggest which CNN features researchers should use in each task.

We describe a system for content-based retrieval from large surveillance video archives, using behavior, action and appearance of objects. Objects are detected, tracked, and classified into broad categories. Their behavior and appearance are characterized by action detectors and descriptors, which are indexed in an archive. Queries can be posed as video exemplars, and the results can be refined through relevance feedback. The contributions of our system include the fusion of behavior and action detectors with appearance for matching; the improvement of query results through interactive query refinement (IQR), which learns a discriminative classifier online based on user feedback; and reasonable performance on low resolution, poor quality video. The system operates on video from ground cameras and aerial platforms, both RGB and IR. Performance is evaluated on publicly-available surveillance datasets, showing that subtle actions can be detected under difficult conditions, with reasonable improvement from IQR.

This paper explores two new aspects of photos and human emotions. First, we show through psychovisual studies that different people have different emotional reactions to the same image, which is a strong and novel departure from previous work that only records and predicts a single dominant emotion for each image. Our studies also show that the same person may have multiple emotional reactions to one image. Predicting emotions in “distributions” instead of a single dominant emotion is important for many applications. Second, we show not only that we can often change the evoked emotion of an image by adjusting color tone and texture related features but also that we can choose in which “emotional direction” this change occurs by selecting a target image. In addition, we present a new database, Emotion6, containing distributions of emotions.

A human can easily find his or her way in an unfamiliar building, by walking around and reading the floor-plan. We try to mimic and automate this human thinking process. More precisely, we introduce a new and useful task of locating an user in the floor-plan, by using only a camera and a floor-plan without any other prior information. We address the problem with a novel matching-localization algorithm that is inspired by human logic. We demonstrate through experiments that our method outperforms state-of-the-art floor-plan-based localization methods by a large margin, while also being highly efficient for real-time applications.

Recent works about convolutional neural networks (CNN) show breakthrough performance on various tasks. However, most of them only use the features extracted from the topmost layer of CNN instead of leveraging the features extracted from different layers. As the first group which explicitly addresses utilizing the features from different layers of CNN, we propose cross-layer CNN features which consist of the features extracted from multiple layers of CNN. Our experimental results show that our proposed cross-layer CNN features outperform not only the state-of-the-art results but also the features commonly used in the traditional CNN framework on three tasks - artistic style, artist, and architectural style classification. As shown by the experimental results, our proposed cross-layer CNN features achieve the best known performance on the three tasks in different domains, which makes our proposed cross-layer CNN features promising solutions for generic tasks.

Most works related to convolutional neural networks (CNN) use the traditional CNN framework which extracts features in only one scale. We propose multi-scale convolutional neural networks (MSCNN) which can not only extract multi-scale features but also solve the issues of the previous methods which use CNN to extract multi-scale features. With the assumption of label-inheritable (LI) property, we also propose a method to generate exponentially more training examples for MSCNN from the given training set. Our experimental results show that MSCNN outperforms both the state-of-the-art methods and the traditional CNN framework on artist, artistic style, and architectural style classification, supporting that MSCNN outperforms the traditional CNN framework on the tasks which at least partially satisfy LI property.

Personalizing image tags is a relatively new and growing area of research, and in order to advance this research community, we must review and challenge the de-facto standard of defining tag importance. We believe that for greater progress to be made, we must go beyond tags that merely describe objects that are visually represented in the image, towards more user-centric and subjective notions such as emotion, sentiment, and preferences. We focus on the notion of user preferences and show that the order that users list tags on images is correlated to the order of preference over the tags that they provided for the image. While this observation is not completely surprising, to our knowledge, we are the first to explore this aspect of user tagging behavior systematically and report empirical results to support this observation. We argue that this observation can be exploited to help advance the image tagging (and related) communities. Our contributions include: 1.) conducting a user study demonstrating this observation, 2.) collecting a dataset with user tag preferences explicitly collected.

Objects occupy physical space and obey physical laws. To truly understand a scene, we must reason about the space that objects in it occupy, and how each objects is supported stably by each other. In other words, we seek to understand which objects would, if moved, cause other objects to fall. This 3D volumetric reasoning is important for many scene understanding tasks, ranging from segmentation of objects to perception of a rich 3D, physically well-founded, interpretations of the scene. In this paper, we propose a new algorithm to parse a single RGB-D image with 3D block units while jointly reasoning about the segments, volumes, supporting relationships, and object stability. Our algorithm is based on the intuition that a good 3D representation of the scene is one that fits the depth data well, and is a stable, self-supporting arrangement of objects (i.e., one that does not topple). We design an energy function for representing the quality of the block representation based on these properties. Our algorithm fits 3D blocks to the depth values corresponding to image segments, and iteratively optimizes the energy function. Our proposed algorithm is the first to consider stability of objects in complex arrangements for reasoning about the underlying structure of the scene. Experimental results show that our stability-reasoning framework improves RGB-D segmentation and scene volumetric representation.

Acute brain diseases such as acute strokes and transit ischemic attacks are the leading causes of mortality and morbidity worldwide, responsible for 9% of total death every year. "Time is brain" is a widely accepted concept in acute cerebrovascular disease treatment. Efficient and accurate computational framework for hemodynamic parameters estimation can save critical time for thrombolytic therapy. Meanwhile the high level of accumulated radiation dosage due to continuous image acquisition in CT perfusion (CTP) raised concerns on patient safety and public health. However, low-radiation leads to increased noise and artifacts which require more sophisticated and time-consuming algorithms for robust estimation. In this paper, we focus on developing a robust and efficient framework to accurately estimate the perfusion parameters at low radiation dosage. Specifically, we present a tensor total-variation (TTV) technique which fuses the spatial correlation of the vascular structure and the temporal continuation of the blood signal flow. An efficient algorithm is proposed to find the solution with fast convergence and reduced computational complexity. Extensive evaluations are carried out in terms of sensitivity to noise levels, estimation accuracy, contrast preservation, and performed on digital perfusion phantom estimation, as well as in vivo clinical subjects. Our framework reduces the necessary radiation dose to only 8% of the original level and outperforms the state-of-art algorithms with peak signal-to-noise ratio improved by 32%. It reduces the oscillation in the residue functions, corrects over-estimation of cerebral blood flow (CBF) and under-estimation of mean transit time (MTT), and maintains the distinction between the deficit and normal regions.

Most works about affective image classification in computer vision treat each emotion category independently and predict hard labels, ignoring the correlation between emotion categories. In this work, inspired by psychological theories, we adopt a dimensional emotion model to model the correlation among certain emotion categories. We also propose a framework of changing image emotion by using our emotion predictor. Easily extendable to other feature transformations, our framework changes image emotion by color histogram specification, relaxing the limitation of the previous method that each emotion is associated with a monotonic palette. Effective and comparable to the previous work of changing image emotion shown by user study, our proposed framework provides users with more flexible control in changing image emotion compared with the previous work.

In this paper, we propose an efficient calibration method to estimate the pointing direction via a human pointing gesture to facilitate robot interaction. The ways in which pointing gestures are used by humans to indicate an object are individually diverse. In addition, people do not always point at the object carefully, which means there is a divergence between the line from the eye to the tip of the index finger and the line of sight. Hence, we focus on adapting to these individual ways of pointing to improve the accuracy of target object identification by means of an effective calibration process. We model these individual ways as two offsets, the horizontal offset and the vertical offset. After locating the head and fingertip positions, we learn these offsets for each individual through a training process with the person pointing at the camera. Experimental results show that our proposed method outperforms other conventional head-hand, head-fingertip, and eye-fingertip-based pointing recognition methods.

3D volumetric reasoning is important for truly understanding a scene. Humans are able to both segment each object in an image, and perceive a rich 3D interpretation of the scene, e.g., the space an object occupies, which objects support other objects, and which objects would, if moved, cause other objects to fall. We propose a new approach for parsing RGB-D images using 3D block units for volumetric reasoning. The algorithm fits image segments with 3D blocks, and iteratively evaluates the scene based on block interaction properties. We produce a 3D representation of the scene based on jointly optimizing over segmentations, block fitting, supporting relations, and object stability. Our algorithm incorporates the intuition that a good 3D representation of the scene is the one that fits the data well, and is a stable, self-supporting (i.e., one that does not topple) arrangement of objects. We experiment on several datasets including controlled and real indoor scenarios. Results show that our stability-reasoning framework improves RGB-D segmentation and scene volumetric representation.

In this work, we consider images of a scene with a moving object captured by a static camera. As the object (human or otherwise) moves about the scene, it reveals pairwise depth-ordering or occlusion cues. The goal of this work is to use these sparse occlusion cues along with monocular depth occlusion cues to densely segment the scene into depth layers. We cast the problem of depth-layer segmentation as a discrete labeling problem on a spatio-temporal Markov Random Field (MRF) that uses the motion occlusion cues along with monocular cues and a smooth motion prior for the moving object. We quantitatively show that depth ordering produced by the proposed combination of the depth cues from object motion and monocular occlusion cues are superior to using either feature independently, and using a naive combination of the features.

Visual attributes are powerful features for many different applications in computer vision such as object detection and scene recognition. Visual attributes present another application that has not been examined as rigorously: verbal communication from a computer to a human. Since many attributes are nameable, the computer is able to communicate these concepts through language. However, this is not a trivial task. Given a set of attributes, selecting a subset to be communicated is task dependent. Moreover, because attribute classifiers are noisy, it is important to find ways to deal with this uncertainty. We address the issue of communication by examining the task of composing an automatic description of a person in a group photo that distinguishes him from the others. We introduce an efficient, principled method for choosing which attributes are included in a short description to maximize the likelihood that a third party will correctly guess to which person the description refers. We compare our algorithm to computer baselines and human describers, and show the strength of our method in creating effective descriptions.

Current works on Information Centric Networking assume the spectrum of caching strategies under the Least Recently/Frequently Used (LRFU) scheme as the de-facto standard, due to the ease of implementation and easier analysis of such strategies. In this paper we predict the popularity distribution of YouTube videos within a campus network. We explore two broad approaches in predicting the popularity of videos in the network: consensus approaches based on aggregate behavior in the network, and social approaches based on the information diffusion over an implicit network. We measure the performance of our approaches under a simple caching framework by picking the k most popular videos according to our predicted distribution and calculating the hit rate on the cache. We develop our approach by first incorporating video inter-arrival time (based on the power-law distribution governing the transmission time between two receivers of the same message in scale-free networks) to the baseline (LRFU), then combining with an information diffusion model over the inferred latent social graph that governs diffusion of videos in the network. We apply techniques from latent social network inference to learn the sharing probabilities between users in the network and apply a virus propagation model borrowed from mathematical epidemiology to estimate the number of times a video will be accessed in the future. Our approach gives rise to a 14% hit rate improvement over the baseline.

In this paper, we propose a novel orientation-aware Urquhart graph based spatial face context representation method to efficiently describe the spatial relationship among faces in group photos. We combine graph matching with orientations of graph edges to assess the similarity of spatial face contexts from different group photos. The experimental results show that our method can find more structurally similar group photos compared to the state-of-the-art spatial face context representation methods.

Most sky models only describe the cloudiness of the overall sky by a single category or parameter such as sky index, which does not account for the distribution of the clouds across the sky. To capture variable cloudiness, we extend the concept of sky index to a random field indicating the level of cloudiness of each sky pixel in our proposed sky representation based on the Igawa sky model. We formulate the problem of solving the sky index of every sky pixel as a labeling problem, where an approximate solution can be efficiently found. Experimental results show that our proposed sky model has better expressiveness, stability with respect to variation in camera parameters, and geo-location estimation in outdoor images compared to the uniform sky index model. Potential applications of our proposed sky model include sky image rendering, where sky images can be generated with an arbitrary cloud distribution at any time and any location, previously impossible with traditional sky models.

In recent years, there has been a great deal of progress in describing objects with attributes. Attributes have proven useful for object recognition, image search, face verification, image description, and zero-shot learning. Typically, attributes are either binary or relative: they describe either the presence or absence of a descriptive characteristic, or the relative magnitude of the characteristic when comparing two exemplars. However, prior work fails to model the actual way in which humans use these attributes in descriptive statements of images. Specifically, it does not address the important interactions between the binary and relative aspects of an attribute. In this work we propose a spoken attribute classifier which models a more natural way of using an attribute in a description. For each attribute we train a classifier which captures the specific way this attribute should be used. We show that as a result of using this model, we produce descriptions about images of people that are more natural and specific than past systems.

We propose a new, challenging, problem in kinship classification: recognizing the family that a query person belongs to from a set of families. We propose a novel framework for recognizing kinship by modeling this problem as that of reconstructing the query face from a mixture of parts from a set of families. To accomplish this, we reconstruct the query face from a sparse set of samples among the candidate families. Our sparse group reconstruction roughly models the biological process of inheritance: a child inherits genetic material from two parents, and therefore may not appear completely similar to either parent, but is instead a composite of the parents. The family classification is determined based on the reconstruction error for each family. On our newly collected “Family101” dataset, we discover links between familial traits among family members and achieve state-of-the-art family classification performance.

To enable high-level understanding of a scene, it is important to understand the occlusion and connected boundaries of objects in the image. In this paper, we propose a new framework for inferring boundaries from color and depth information. Even with depth information, it is not a trivial task to find and classify boundaries. Real-world depth images are noisy, especially at object boundaries, where our task is focused. Our approach uses features from both the color (which are sharp at object boundaries) and depth images (for providing geometric cues) to detect boundaries and classify them as occlusion or connected boundaries. We propose depth features based on surface fitting from sparse point clouds, and perform inference with a Conditional Random Field. One advantage of our approach is that occlusion and connected boundaries are identified with a single, common model. Experiments show that our mid-level color and depth features outperform using either depth or color alone, and our method surpasses the performance of baseline boundary detection methods.

Photography on a mobile camera provides access to additional sensors. In this paper, we estimate the absolute orientation of a planar object with respect to the ground, which can be a valuable prior for many vision tasks. To find the planar object orientation, our novel algorithm combines information from a gravity sensor with a planar homography that matches a region of an image to a training image (e.g., of a company logo). We demonstrate our approach with an iPhone application that records the gravity direction for each captured image. We find a homography that maps the training image to the test image, and propose a novel homography decomposition to extract the rotation matrix. We believe this is the first paper to estimate absolute planar object orientation by combining the inertial sensor information with vision algorithms. Experiments show that our proposed algorithm performs reliably.

Microsoft's Picture Password provides a method to authenticate a user without the need of typing a character based password. The password consists of a set of gestures drawn on an image. The position, direction and order of these gestures constitute the password. Besides being more convenient to use on touch screen devices, this authentication method promises improved memorability in addition to improving the password strength against guessing attacks. However, how unpredictable is the picture password? In this paper we exploit the fact that different users are drawn to similar image regions, and therefore these passwords are vulnerable to guessing attacks. More specifically, we show that for portrait pictures users are strongly drawn to use facial features as gesture locations. We collect a set of Picture Passwords and, using computer vision techniques, derive a list of password guesses in decreasing probability order. We show that guessing in this order we are able to improve the likelihood of cracking a password within a limited number of guesses.

When people gather for a group photo, they are together for a social reason. Past work has shown that these social relationships affect how people position themselves in a group photograph. We propose classifying the type of group photo based on the spatial arrangement and the predicted attributes of the faces in the image. We propose a matching algorithm for finding images from a training set that have both similar arrangement of faces and attribute correspondence. We formulate the problem as a bipartite matching problem where the faces from each of the pair of images are nodes in the graph. Our work demonstrates that face arrangement, when combined with attribute (age and gender) correspondence, is a useful cue in capturing an approximate social essence of the group of people, and lets us understand why the group of people gathered for the photo.

We propose a facial trait code (FTC) to encode human facial images, and apply it to face recognition. Extracted from an exhaustive set of local patches cropped from a large stack of faces, the facial traits and the associated trait patterns can accurately capture the appearance of a given face. The extraction has two phases. The first phase is composed of clustering and boosting upon a training set of faces with neutral expression, even illumination, and frontal pose. The second phase focuses on the extraction of the facial trait patterns from the set of faces with variations in expression, illumination, and poses. To apply the FTC to face recognition, two types of codewords, hard and probabilistic, with different metrics for characterizing the facial trait patterns are proposed. The hard codeword offers a concise representation of a face, while the probabilistic codeword enables matching with better accuracy. Our experiments compare the proposed FTC to other algorithms on several public datasets, all showing promising results.

The framework of capturing multiview images has received a great attention from industry as well as academy due to its usability in various applications; 3D scene reconstruction, super-resolution, scene understanding, surveillance etc. This paper presents a multiview capturing system that uses a single camera with multiple mirrors. We place multiple mirrors pointing to the target scene and record an image of those mirrors. From this image of mirrors, we can extract the multiview images of the target scene, images reflected by mirrors. Based on the proposed method, we achieve some success in geometry reconstruction and view synthesis using multiview images, which shows the usability of the proposed work. The main contribution of this work is to introduce a light weight implementation for multiview capturing with a number of useful properties; the automatic synchronization across multiple views, the consistent color/white balance setting across multiple views, the capability of recording a wide range of view and the cost-effective implementation.

Depth ordering is instrumental for understanding the 3D geometry of an image. Humans are surprisingly good at depth ordering even with abstract 2D line drawings. In this paper we propose a learning-based framework for depth ordering inference. Boundary and junction characteristics are important clues for this task, and we have developed new features based on these attributes. Although each feature individually can produce reasonable depth ordering results, each still has limitations, and we can achieve better performance by combining them. In practice, local depth ordering inferences can be contradictory. Therefore, we propose a Markov Random Field model with terms that are more global than previous work, and use graph optimization to encourage a globally consistent ordering. In addition, to produce better object segmentation for the task of depth ordering, we propose to explicitly enforce closed loops and long edges for the occlusion boundary detection. We collect a new depth-order dataset for this problem, including more than a thousand human-labeled images with various daily objects and configurations. The proposed algorithm shows promising performance over conventional methods on both synthetic and real scenes.

Objects in scenes interact with each other in complex ways. A key observation is that these interactions manifest themselves as predictable visual patterns in the image. Discovering and detecting these structured patterns is an important step towards deeper scene understanding. It goes beyond using either individual objects or the scene as a whole as the semantic unit. In this work, we promote "groups of objects". They are high-order composites of objects that demonstrate consistent spatial, scale, and viewpoint interactions with each other. These groups of objects are likely to correspond to a specific layout of the scene. They can thus provide cues for the scene category and can also prime the likely locations of other objects in the scene. It is not feasible to manually generate a list of all possible groupings of objects we find in our visual world. Hence, we propose an algorithm that automatically discovers groups of arbitrary numbers of participating objects from a collection of images labeled with object categories. Our approach builds a 4-dimensional transform space of location, scale and viewpoint, and efficiently identifies all recurring compositions of objects across images. We then model the discovered groups of objects using the deformable parts-based model. Our experiments on a variety of datasets show that using groups of objects can significantly boost the performance of object detection and scene categorization.

Many works in computer vision attempt to solve different tasks such as object detection, scene recognition or attribute detection, either separately or as a joint problem. In recent years, there has been a growing interest in combining the results from these different tasks in order to provide a textual description of the scene. However, when describing a scene, there are many items that can be mentioned. If we include all the objects, relationships, and attributes that exist in the image, the description would be extremely long and not convey a true understanding of the image. We present a novel approach to ranking the importance of the items to be described. Specifically, we focus on the task of discriminating one image from a group of others. We investigate the factors that contribute to the most efficient description that achieves this task. We also provide a quantitative method to measure the description quality for this specific task using data from human subjects and show that our method achieves better results than baseline methods.

The Conditional Random Field (CRF) is a popular tool for object-based image segmentation. CRFs used in practice typically have edges only between adjacent image pixels. To represent object relationship statistics beyond adjacent pixels, prior work either represents only weak spatial information using the segmented regions, or encodes only global object co-occurrences. In this paper, we propose a unified model that augments the pixel-wise CRFs to capture object spatial relationships. To this end, we use a fully connected CRF, which has an edge for each pair of pixels. The edge potentials are defined to capture the spatial information and preserve the object boundaries at the same time. Traditional inference methods, such as belief propagation and graph cuts, are impractical in such a case where billions of edges are defined. Under only one assumption that the spatial relationships among different objects only depend on their relative positions (spatially stationary), we develop an efficient inference algorithm that converges in a few seconds on a standard resolution image, where belief propagation takes more than one hour for a single iteration.

A camera free 3D interactive system for multi-user/multi-touch was proposed. By embedding the optical sensors in display pixels, the flat panel can sense images projected by infrared(IR) light. A light pen which can project T-mark IR image was utilized to determine the 5-axis (x, y, z, θ, φ) information of the user. Additionally, multiple T-marks with sequentially illuminating method can further identify multi-users even the marks are overlapped. Therefore, a 3-dimensional virtual touch display for 5-axis information with multi-user system can be achieved.

Given a video of a dynamic scene captured using a dynamic camera, we present a method to recover a dense depth map of the scene with a focus on estimating the depth of the dynamic objects. We assume that the static portions of the scene help estimate the pose of the cameras. We recover a dense depth map of the scene via a plane sweep stereo approach. The relative motion of the dynamic object in the scene however, results in an inaccurate depth estimate. Estimating the accurate depth of the dynamic object is an ambiguous problem since both the depth and the real world speed of the object are unknown. In this work, we show that by using occlusions and putting constraints on the speed of the object we can bound the depth of the object. We can then incorporate this real world motion into the plane sweep stereo framework to obtain a more accurate depth for the dynamic object. We focus on videos with people walking in the scene and show the effectiveness of our approach through quantitative and qualitative results.

The hierarchical structures that exist in natural scenes have been utilized for many tasks in computer vision. The basic idea is that instead of using strictly low level features it is possible to combine them into higher level hierarchical structures. These higher level structures provide a more specific feature and can thus lead to better results in classification or detection. Although most previous work has focused on hierarchical combinations of low level features, hierarchical structures exist on higher levels as well. In this work we attempt to automatically discover these higher level structures by finding meaningful object groups using the Minimum Description Length (MDL) principle. We then use these structures for scene classification and show that we can achieve a higher accuracy rate using them.

We propose a novel method for the problem of target retrieval for wide area video surveillance (WAVS). Different from previous methods, our algorithm does not require environment model construction nor the learning or usage of any transition probabilities. Furthermore, our method can be applied to an arbitrarily large area for surveillance. The essence of our method lies in an optimization problem formulation using only the shortest driving distance between pairs of cameras, and the provision of an algorithm that finds a global optimum to the optimization. Experiments show that our method can achieve 99% precision and > 80% recall on an input of thousands of frames, which far exceeds the performance of the baseline, whilst taking under 10 seconds to run on a 1.67GHz Intel Centrino laptop. This makes it feasibly deployable on handheld devices in a police car.

Recently, a novel "completely automated public Turing test to tell computers and humans apart (CAPTCHA)'' system has been proposed, in which users are asked to separate natural faces of humans and artificial faces of virtual world avatars. The system is based on the assumption that computers cannot separate them while it is an easy task for humans. Conventional digital forensics approaches to distinguish natural images from computer graphics images are mostly based on statistical analysis of the images such as noise in CMOS image sensors or Bayer matrix estimation. On the other hand, this paper uses face recognition and object classification based approaches. The experiments show that our approaches work surprisingly well and yields more than 99% accuracy. Our object classification based approach can also tell us how likely the input images are regarded as human/avatar faces.

In this paper we present a framework to obtain highly efficient implementations for the narrow band level set method on commercial off-the-shelf (COTS) multicore CPU systems with a cache-based memory hierarchy such as Intel Xeon and Atom processors. The narrow-band level set algorithm tracks wave-fronts in discretized volumes (for instance, explosion shock waves), and is computationally very demanding. At the core of our optimization framework is a novel projection-based approach to enhance data locality and enable reuse for sparse surfaces in dense discretized volumes. The method reduces stencil operations on sparse and changing sets of pixels belonging to an evolving surface into dense stencil operations on meta-pixels in a lower-dimensional projection of the pixel space. These meta-pixels are then amenable to standard techniques like time tiling. However, the complexity introduced by ever-changing meta-pixels requires us to revisit and adapt all other necessary optimizations. We apply adapted versions of SIMDization, multi-threading, DAG scheduling for basic tiles, and specialization through code generation to extract maximum performance. The system is implemented as highly parameterized code skeleton that is auto-tuned and uses program generation. We evaluated our framework on a dual-socket 2.8 GHz Xeon 5560 and a 1.6 GHz Atom N270. Our single-core performance reaches 26%-35% of the machine peak on the Xeon, and 12%- 20% on the Atom across a range of image sizes. We see up to 6.5x speedup on 8 cores of the dual-socket Xeon. For cacheresident sizes our code outperforms the best available thirdparty code (C pre-compiled into a DLL) by about 10x and for the largest out-of-cache sizes the speedup approaches around 200x. Experiments fully explain the high speedup numbers.

Scene understanding includes many related subtasks, such as scene categorization, depth estimation, object detection, etc. Each of these subtasks is often notoriously hard, and state-of-the-art classifiers already exist for many of them. These classifiers operate on the same raw image and provide correlated outputs. It is desirable to have an algorithm that can capture such correlation without requiring any changes to the inner workings of any classifier. We propose Feedback Enabled Cascaded Classification Models (FE-CCM), that jointly optimizes all the subtasks while requiring only a "black box" interface to the original classifier for each subtask. We use a two-layer cascade of classifiers, which are repeated instantiations of the original ones, with the output of the first layer fed into the second layer as input. Our training method involves a feedback step that allows later classifiers to provide earlier classifiers information about which error modes to focus on. We show that our method significantly improves performance in all the subtasks in the domain of scene understanding, where we consider depth estimation, scene categorization, event categorization, object detection, geometric labeling, and saliency detection. Our method also improves performance in two robotic applications: an object-grasping robot and an object-finding robot.

Typically, object recognition is performed based solely on the appearance of the object. However, relevant information also exists in the scene surrounding the object. In this paper, we explore the roles that appearance and contextual information play in object recognition. Through machine experiments and human studies, we show that the importance of contextual information varies with the quality of the appearance information, such as an image's resolution. Our machine experiments explicitly model context between object categories through the use of relative location and relative scale, in addition to co-occurrence. With the use of our context model, our algorithm achieves state-of-the-art performance on the MSRC and Corel data sets. We perform recognition tests for machines and human subjects on low and high resolution images, which vary significantly in the amount of appearance information present, using just the object appearance information, the combination of appearance and context, as well as just context without object appearance information (blind recognition). We also explore the impact of the different sources of context (co-occurrence, relative-location, and relative-scale). We find that the importance of different types of contextual information varies significantly across data sets such as MSRC and PASCAL.

This paper presents an efficient opportunistic retransmission protocol (PRO, Protocol for Retransmitting Opportunistically) to improve the performance of IEEE 802.11 WLANs. PRO is a link-layer protocol that allows overhearing nodes to function as relays that retransmit on behalf of a source after they learn about a failed transmission. Relays with better connectivity to the destination have a higher chance of delivering the packet than the source, thereby resulting in a more efficient use of the channel. PRO has four main features. First, channel reciprocity coupled with a runtime calibration process is used to estimate the instantaneous link quality to the destination. Second, a local qualification process filters out poor relays early. Third, a distributed relay selection algorithm chooses the best set of eligible relays among all qualified relays and prioritizes them. Finally, 802.11e Enhanced Distributed Channel Access (EDCA) is leveraged to make sure high-quality relays transmit with higher probability. PRO is designed to coexist with legacy 802.11 stations. Our extensive evaluation on both a controlled testbed and in the real world shows that PRO can improve throughput in diverse wireless environments. PRO helps the most when there is significant contention for the ether, under fading, and with user mobility.

This paper presents an active-learning algorithm for piecewise planar 3D reconstruction of a scene. While previous interactive algorithms require the user to provide tedious interactions to identify all the planes in the scene, we build on successful ideas from the automatic algorithms and introduce the idea of active learning, thereby improving the reconstructions while considerably reducing the effort. Our algorithm first attempts to obtain a piecewise planar reconstruction of the scene automatically through an energy minimization framework. The proposed active-learning algorithm then uses intuitive cues to quantify the uncertainty of the algorithm and suggest regions, querying the user to provide support for the uncertain regions via simple scribbles. These interactions are used to suitably update the algorithm, leading to better reconstructions. We show through machine experiments and a user study that the proposed approach can intelligently query users for interactions on informative regions, and users can achieve better reconstructions of the scene faster, especially for scenes with texture-less surfaces lacking cues like lines which automatic algorithms rely on.

The most popular approach to large scale image retrieval is based on the bag-of-visual-word (BoV) representation of images. The spatial information is usually re-introduced as a post-processing step to re-rank the retrieved images, through a spatial verification like RANSAC. Since the spatial verification techniques are computationally expensive, they can be applied only to the top images in the initial ranking. In this paper, we propose an approach that can encode more spatial information into BoV representation and that is efficient enough to be applied to large-scale databases. Other works pursuing the same purpose have proposed exploring the word co-occurrences in the neighborhood areas. Our approach encodes more spatial information through the geometry-preserving visual phrases (GVP). In addition to co-occurrences, the GVP method also captures the local and long-range spatial layouts of the words. Our GVP based searching algorithm increases little memory usage or computational time compared to the BoV method. Moreover, we show that our approach can also be integrated to the min-hash method to improve its retrieval accuracy. The experiment results on Oxford 5K and Flicker 1M dataset show that our approach outperforms the BoV method even following a RANSAC verification.

Images taken from scenes under water suffer distortion due to refraction. While refraction causes magnification with mild distortion on the observed images, severe distortions in geometry reconstruction would be resulted if the refractive distortion is not properly handled. Different from the radial distortion model, the refractive distortion depends on the scene depth seen from each light ray as well as the camera pose relative to the refractive surface. Therefore, it's crucial to obtain a good estimate of scene depth, camera pose and optical center to alleviate the impact of refractive distortion. In this work, we formulate the forward and back projections of light rays involving a refractive plane for the perspective camera model by explicitly modeling refractive distortion as a function of depth. Furthermore, for cameras with an inertial measurement unit (IMU), we show that a linear solution to the relative pose and a closed-form solution to the absolute pose can be derived with known camera vertical directions. We incorporate our formulations with the general structure from motion framework followed by the patch-based multiview stereo algorithm to obtain a 3D reconstruction of the scene. We show through experiments that the explicit modeling of depth-dependent refractive distortion physically leads to more accurate scene reconstructions.

Existing approaches to contextual reasoning for enhanced object detection typically utilize other labeled categories in the images to provide contextual information. As a consequence, they inadvertently commit to the granularity of information implicit in the labels. Moreover, large portions of the images may not belong to any of the manually-chosen categories, and these unlabeled regions are typically neglected. In this paper, we overcome both these drawbacks and propose a contextual cue that exploits unlabeled regions in images. Our approach adaptively determines the granularity (scene, inter-object, intra-object, etc.) at which contextual information is captured. In order to extract the proposed contextual cue, we consider a scene to be a structured configuration of objects and regions; just as an object is a composition of parts. We thus learn our proposed “contextual meta-objects” using any off-the-shelf object detector, which makes our proposed cue widely accessible to the community. Our results show that incorporating our proposed cue provides a relative improvement of 12% over a state-of-the-art object detector on the challenging PASCAL dataset.

Semi-supervised learning uses both labeled and unlabeled data for machine learning tasks. It's especially useful in the scenarios where labeled data is very scarce or expensive to obtain. In this work, we present kernel LLC, the kernel locality-constrained linear coding within a data-dependent kernel space, for data representation. The data-dependent kernel captures the underlying data geometry on the ambient feature space. The kernel LLC further exploits the locality association among the data on its manifold. Promising results on both image classification and content-based image retrieval scenarios suggest kernel LLC to be a good candidate for data representation in semi-supervised learning.

A laser scan is useful in building the 3D model, and in one ran it can capture thousands of 3D points. However these 3D points are sparse compared to a normal image, which can easily have millions of pixels. To achieve a denser 3D map, 3D-interpolation is applied to each pixel in the image. In this work we propose an algorithm to combine the 3D geometry and the color for 3D-interpolation. We segment the 3D points based on their latent surfaces, and combine the surfaces with color through Markov Random Field. We find that the 3D geometry provides rich information for interpolation: 3D points with similar colors can be robustly clustered where not possible in the color space, and the interpolation can be performed on a better fitting surface rather than on the locally linear ones. Our experiments show that the proposed algorithm outperforms the baselines.

In this paper, we present a novel interactive 3D reconstruction algorithm which renders a planar reconstruction of the scene. We consider a scenario where the user has taken a few images of a scene from multiple poses. The goal is to obtain a dense and visually pleasing reconstruction of the scene, including non-planar objects. Using simple user interactions in the form of scribbles indicating the surfaces in the scene, we develop an idea of 3D scribbles to propagate scene geometry across multiple views and perform co-segmentation of all the images into the different surfaces and non-planar objects in the scene. We show that this allows us to render a complete and pleasing reconstruction of the scene along with a volumetric rendering of the non-planar objects. We demonstrate the effectiveness of our algorithm on both outdoor and indoor scenes including the ability to handle featureless surfaces.

Detecting foreground objects for night surveillance videos remains a challenging problem in scene understanding. Though many efforts have been made for robust background subtraction and robust object detection respectively, the complex illumination condition in night scenes makes it hard to solve each of these tasks individually. In practice, we see these two tasks are coupled and can be combined to help each other. In this work, we apply a recently proposed algorithm - Feedback Enabled Cascaded Classification Models (FECCM) - to combine the background subtraction task and the object detection task into a generic framework. The proposed framework treats each classifier for the respective task as a `black-box', thus allows the usage of most existing algorithms as one of the classifiers. Experiment results show that the proposed method outperforms a state-of-the-art background subtraction method and a state-of-the-art object detection method.

Although the sketch recognition and computer vision communities attempt to solve similar problems in different domains, the sketch recognition community has not utilized many of the advancements made in computer vision algorithms. In this paper we propose using a pictorial structure model for object detection, and modify it to better perform in a drawing setting as opposed to photographs. By using this model we are able to detect a learned object in a general drawing, and correctly label its parts. We show our results on 4 categories.

We address the problem of automatic color categorization of the objects in surveillance videos. This problem is challenging for realistic situations due to the large intra-class variations of the same color and the large portions of noisy areas including the backgrounds and the parts of the objects that do not contribute to color assignments. We develop an integrated color categorization system with algorithms that address these challenges. With the algorithms proposed in this paper, we can improve the average color categorization accuracy by 18% from our previous work [7].

Object detection is a basic skill for a robot to perfor-m tasks in human environments. In order to build a good object classifier, a large training set of la-beled images is required; this is typically collect-ed and labeled (often painstakingly) by a human. This method is not scalable and therefore limits the robot’s detection performance. We propose an algorithm for a robot to collect more data in the environment during its training phase so that in the future it could detect objects more reli-ably. The first step is to plan a path for collecting additional training images, which is hard because a previously visited location affects the decision for the future locations. One key component of our work is path planning by building a sparse graph that captures these dependencies. The other key component is our learning algorithm that weighs the errors made in robot’s data collection process while updating the classifier. In our experiments, we show that our algorithms enable the robot to im-prove its object classifiers significantly.

For most scene understanding tasks (such as object detection or depth estimation), the classifiers need to consider contextual information in addition to the local features. We can capture such contextual information by taking as input the features/attributes from all the regions in the image. However, this contextual dependence also varies with the spatial location of the region of interest, and we therefore need a different set of parameters for each spatial location. This results in a very large number of parameters. In this work, we model the independence properties between the parameters for each location and for each task, by defining a Markov Random Field (MRF) over the parameters. In particular, two sets of parameters are encouraged to have similar values if they are spatially close or semantically close. Our method is, in principle, complementary to other ways of capturing context such as the ones that use a graphical model over the labels instead. In extensive evaluation over two different settings, of multi-class object detection and of multiple scene understanding tasks (scene categorization, depth estimation, geometric labeling), our method beats the state-of-the-art methods in all the four tasks.

In this paper, we study performance optimization for scalable video coding and multicast over networks. Multi-path video streaming, network coding based routing, and network flow control are jointly optimized to maximize a network utility function defined over heterogeneous receivers. Content priority of video coding layers is considered during the flow routing to determine the optimal multicast paths and associated data rates for each layer. Our optimization scheme attempts to find content distribution meshes with minimum path costs for each video coding layer while satisfying the inter-layer dependency during scalable video coding. Based on primal decomposition and primal-dual analysis, we develop a decentralized algorithm with two optimization levels to solve the performance optimization problem. We also prove the stability and convergence of the proposed iterative algorithm using Lyapunov theories. Extensive experimental results demonstrate that the proposed algorithm not only achieves the max-flow throughput using network coding, but also provides better video quality with balanced layered access for heterogeneous receivers.

10.5244/C.24.46

10.5244/C.24.47

10.5244/C.24.105

In this paper we propose a framework for gradient descent image alignment in the Fourier domain. Specifically, we propose an extension to the classical Lucas & Kanade (LK) algorithm where we represent the source and template image's intensity pixels in the complex 2D Fourier domain rather than in the 2D spatial domain. We refer to this approach as the Fourier LK (FLK) algorithm. The FLK formulation is especially advantageous, over traditional LK, when it comes to pre-processing the source and template images with a bank of filters (e.g., Gabor filters) as: (i) it can handle substantial illumination variations, (ii) the inefficient pre-processing filter bank step can be subsumed within the FLK algorithm as a sparse diagonal weighting matrix, (iii) unlike traditional LK the computational cost is invariant to the number of filters and as a result far more efficient, (iv) this approach can be extended to the inverse compositional form of the LK algorithm where nearly all steps (including Fourier transform and filter bank pre-processing) can be pre-computed leading to an extremely efficient and robust approach to gradient descent image matching. We demonstrate robust image matching performance on a variety of objects in the presence of substantial illumination differences with exactly the same computational overhead as that of traditional inverse compositional LK during fitting.

Maximum a posteriori (MAP) inference in Markov Random Fields (MRFs) is an NP-hard problem, and thus research has focussed on either finding efficiently solvable subclasses (e.g. trees), or approximate algorithms (e.g. Loopy Belief Propagation (BP) and Tree-reweighted (TRW) methods). This paper presents a unifying perspective of these approximate techniques called "Decomposition Methods". These are methods that decompose the given problem over a graph into tractable subproblems over subgraphs and then employ message passing over these subgraphs to merge the solutions of the subproblems into a global solution. This provides a new way of thinking about BP and TRW as successive steps in a hierarchy of decomposition methods. Using this framework, we take a principled first step towards extending this hierarchy beyond trees. We leverage a new class of graphs amenable to exact inference, called outer-planar graphs, and propose an approximate inference algorithm called Outer-Planar Decomposition (OPD). OPD is a strict generalization of BP and TRW, and contains both of them as special cases. Our experiments show that this extension beyond trees is indeed very powerful -OPD outperforms current state-of-art inference methods on hard non-submodular synthetic problems and is competitive on real computer vision applications.

This paper presents an algorithm for Interactive Co-segmentation of a foreground object from a group of related images. While previous approaches focus on unsupervised co-segmentation, we use successful ideas from the interactive object-cutout literature. We develop an algorithm that allows users to decide what foreground is, and then guide the output of the co-segmentation algorithm towards it via scribbles. Interestingly, keeping a user in the loop leads to simpler and highly parallelizable energy functions, allowing us to work with significantly more images per group. However, unlike the interactive single image counterpart, a user cannot be expected to exhaustively examine all cutouts (from tens of images) returned by the system to make corrections. Hence, we propose iCoseg, an automatic recommendation system that intelligently recommends where the user should scribble next. We introduce and make publicly available the largest co-segmentation datasetyet, the CMU-Cornell iCoseg Dataset, with 38 groups, 643 images, and pixelwise hand-annotated groundtruth. Through machine experiments and real user studies with our developed interface, we show that iCoseg can intelligently recommend regions to scribble on, and users following these recommendations can achieve good quality cutouts with significantly lower time and effort than exhaustively examining all cutouts.

In using image analysis to assist a driver to avoid obstacles on the road, traditional approaches rely on various detectors designed to detect different types of objects. We propose a framework that is different from traditional approaches in that it focuses on finding a clear path ahead. We assume that the video camera is calibrated offline (with known intrinsic and extrinsic parameters) and vehicle information (vehicle speed and yaw angle) is known. We first generate perspective patches for feature extraction in the image. Then, after extracting and selecting features of each patch, we estimate an initial probability that the patch corresponds to clear path using a support vector machine (SVM) based probability estimator on the selected features. We finally perform probabilistic patch smoothing based on spatial and temporal constraints to improve the initial estimate, thereby enhancing detection performance. We show that the proposed framework performs well even in some challenging examples with shadows and illumination changes.

In this paper, we describe a prior-based vanishing point estimation method through global perspective structure matching (GPSM). In contrast to the traditional approaches which require an undistorted image with straight roads for vanishing point estimation, our method first infers vanishing point candidates of an input image from an image database with pre-labeled vanishing points. An image-based retrieval method is used to identify the best candidate images in the database by matching image's perspective structure. The initial estimation of input image's vanishing point is calculated from the pre-labeled vanishing points of the best candidates. Probabilistic refinement (PR) is then used to optimize the vanishing point estimate. Experimental results show that the proposed method works well in a variety of on-road driving environments (e.g., in urban, highway and country-side areas), especially with traffic captured by a fish-eye back-aid camera.

This paper aims to optimize joint overall video quality and traffic performance of multirate multicast of scalable video streaming accessing hybrid wired/wireless paths. In order to guarantee layered utility maximization over tiered wired/wireless coded networks, we propose a joint source and network flow optimization scheme where each scalable layer is tailored in an incremental order and finds jointly optimal multicast paths and associated rates. It not only achieves the highest sustainable layered video quality, but also takes into account both path-overlapping allocation for different receivers which takes advantage of network coding and the wireless link contention in a shared transmission medium. The convex programming is imposed on the total variation minimization of layered rate-distortion. Using primal decomposition and the primal-dual approach, we develop a decentralized algorithm with two levels of optimization. Experimental results demonstrate that the proposed algorithm not only provides better video quality with optimal layered access for heterogeneous receivers, but benefits the asymptotic capacity of a hierarchical "hybrid" wired/wireless architecture in a scalable sense.

In this paper, we propose a three-stage method for annotating image groups in real-world photo image databases: (1) Image databases are automatically divided into several image groups so that most photos in each group were taken in the same scene using the time information associated with each photo; (2) Objects in each image are recognized using multiclass object recognition; (3) Each image group is categorized into a scene using all object labels from (2) in the image group. Our main contribution is to propose a novel method for annotating image groups using all objects recognized from all images in this image group. We train our method on 6,000 objects in 696 images from the LabelMe dataset and verify the effectiveness of our proposed method on real-world photo databases consists of 4 outdoor scenes.

Segmentation-based approach has shown significant success in stereo matching. By assuming pixels within one image segment belong to the same 3D surface, robust depth estimation can be achieved by taking the whole segment into consideration. However, segmentation has been mostly used for stereo matching at integer disparities rather than subpixel disparities. One major reason is that small segments may be insufficient for estimating surfaces like slanted planes, while large segments may contain segmentation errors impacting the accuracy of depth estimation. In this work, we propose a segmentation-based scheme for subpixel stereo matching. Instead of using a fixed segmentation, segments are evolved to find a better support for reliable surface estimation. Given an initial estimation of segmentation and depth, the proposed algorithm jointly optimizes the segmentation and depth by evolving the segmentation at the pixel level and updating the plane parameters at the segment level. Justified with experiments performed on the Middlebury benchmark, we show that the proposed method achieves significant improvements for subpixel stereo matching.

We tackle the challenge of kinship verification using novel feature extraction and selection methods, automatically classifying pairs of face images as “related” or “unrelated” (in terms of kinship). First, we conducted a controlled online search to collect frontal face images of 150 pairs of public figures and celebrities, along with images of their parents or children. Next, we propose and evaluate a set of low-level image features for this classification problem. After selecting the most discriminative inherited facial features, we demonstrate a classification accuracy of 70.67% on a test set of image pairs using K-Nearest-Neighbors. Finally, we present an evaluation of human performance on this problem.

Object of Interest (OOI) detection has been widely used in many recent works in video analysis, especially in video similarity and video retrieval. In this paper, we describe a generic video classification algorithm using object of interest detection. We use online user-submitted videos and aim to categorize the videos into six broad categories hot star, news, anime, pets, sports and commercials. We show through our experiments that, detecting and describing the object of interest improves the video classification accuracy by about 10 percentage points.

Automatically assessing the subjective quality of a photo is a challenging area in visual computing. Previous works study the aesthetic quality assessment on a general set of photos regardless of the photo's content and mainly use features extracted from the entire image. In this work, we focus on a specific genre of photos: consumer photos with faces. This group of photos constitutes an important part of consumer photo collections. We first conduct an online study on Mechanical Turk to collect ground-truth and subjective opinions for a database of consumer photos with faces. We then extract technical features, perceptual features, and social relationship features to represent the aesthetic quality of a photo, by focusing on face-related regions. Experiments show that our features perform well for categorizing or predicting the aesthetic quality.

In this paper, we consider the problem of automatic landmark image recognition. Specifically, we identify a fundamental issue that lurks in such applications as modern landmark recognition that arises as a natural consequence of a current state-of-the-art techinque, namely one-versus-all SVM. Then, we provide a unary classification approach that retains much of the benefits of one-versus-all SVM's whilst avoids their shortcomings in the context of landmark recognition tasks. Finally, we provide empirical evidence of the improvements based on our experiments.

In this paper, we propose the use of 3D information to augment the Markov random field (MRF) model for object recognition. Conventional MRF for image-based object recognition usually uses appearance and 2D location as features in the model. We estimate rough 3D information from stereo image pairs, and incorporate this information into node and edge potential models in the conventional MRF. Introducing 3D information into the node potential allows to leverage the distribution statistics of 3D location for different classes. We solve the object recognition problem by finding the globally optimal class assignment that minimizes an energy function defined in the augmented MRF. We show that the introduction of 3D distance in the edge potential can help distinguish “true” neighbors from “fake” neighbors in 2D. We demonstrate improved recognition results by using the proposed technique.

Active models are widely used in applications like image segmentation and tracking. Region-based active models are known for robustness to weak edges and high computational complexity. We found previous region-based models can easily get stuck in local minimums if initialization is far from the true object boundary. This is caused by an inherent ambiguity in evolution direction of the level set function when minimizing the energy. To solve this problem, we propose an intensity re-weighting (IR) model to bias the evolution process in certain direction. IR model can effectively avoid local minimums and enable much faster convergence of the evolution process. The proposed method is applied to both real and synthetic images with promising results.

Direct implementations of bilateral filtering show O(r2) computational complexity per pixel, where r is the filter window radius. Several lower complexity methods have been developed. State-of-the-art low complexity algorithm is an O(1) bilateral filtering, in which computational cost per pixel is nearly constant for large image size. Although the overall computational complexity does not go up with the window radius, it is linearly proportional to the number of quantization levels of bilateral filtering computed per pixel in the algorithm. In this paper, we show that overall runtime depends on two factors, computing time per pixel per level and average number of levels per pixel. We explain a fundamental trade-off between these two factors, which can be controlled by adjusting block size. We establish a model to estimate run time and search for the optimal block size. Using this model, we demonstrate an average speedup of 1.2-26.0x over the previous method for typical bilateral filtering parameters.

We explore the problem of object color categorization from natural images. Previous works use the histograms of RGB values of images with learning base methods. We propose to use shape information to help to localize the foreground areas of an image that determine the color of the object (such as car hoods), and focus the color learning and prediction on these areas. A novel Co-PLSA model is proposed to jointly learn the color and shape detectors in weakly supervised manner, where training images are only labeled with the color categories, while the locations of the foreground areas are not provided.

Automatic photo quality assessment and selection systems are helpful for managing the large mount of consumer photos. In this paper, we present such a system based on evaluating the aesthetic quality of consumer photos. The proposed system focuses on photos with faces, which constitute an important part of consumer photo albums. The system has three contributions: 1) We propose an aesthetics-based photo assessment algorithm, by considering different aesthetics-related factors, including the technical characteristics of the photo and the specific features related to faces; 2) Based on the aesthetic measurement, we propose a cropping-based photo editing algorithm, which differs from prior works by eliminating unimportant faces before optimizing photo composition; 3) We also incorporate the aesthetic evaluation with other metrics to select quintessential photos for a large collection of photos. The entire system is delivered by a web interface, which allows users to submit images or albums, and returns promising results for photo evaluation, editing recommendation, and photo selection.

In many machine learning domains (such as scene understanding), several related sub-tasks (such as scene categorization, depth estimation, object detection) operate on the same raw data and provide correlated outputs. Each of these tasks is often notoriously hard, and state-of-the-art classifiers already exist for many sub-tasks. It is desirable to have an algorithm that can capture such correlation without requiring to make any changes to the inner workings of any classifier. We propose Feedback Enabled Cascaded Classification Models (FE-CCM), that maximizes the joint likelihood of the sub-tasks, while requiring only a ‘black-box’ interface to the original classifier for each sub-task. We use a two-layer cascade of classifiers, which are repeated instantiations of the original ones, with the output of the first layer fed into the second layer as input. Our training method involves a feedback step that allows later classifiers to provide earlier classifiers information about what error modes to focus on. We show that our method significantly improves performance in all the sub-tasks in two different domains: (i) scene understanding, where we consider depth estimation, scene categorization, event categorization, object detection, geometric labeling and saliency detection, and (ii) robotic grasping, where we consider grasp point detection and object classification.

Bandwidth demands and timing constraints are two major challenges for wireless video streaming applications. In this paper, we present a hybrid spatial/temporal retransmission protocol that tackles both of these challenges. To increase individual throughput as well as overall network capacity, the system uses an opportunistic retransmission protocol (PRO, Protocol for Retransmitting Opportunistically) that relies on overhearing nodes distributed in physical space to function as relays that retransmit failed packets on behalf of the source. Specifically, the best relay out of the set of nodes that currently have the copy of the packet is responsible for retransmitting (relaying) the packet. Relays with stronger connectivity to the destination have a higher chance of delivering packets successfully than the source, thereby resulting in a more efficient use of the channel. To meet timing constraints, a Time-based Adaptive Retransmission strategy (TAR) is applied by both the source and the relays. With TAR, the MAC dynamically determines whether to (re)transmit or discard a packet based on the retransmission deadline of the packet assigned by the video server. This significantly reduces the number of late packet arrivals at the receiver. Our extensive evaluation results both on a testbed and in the real world demonstrate that hybrid temporal/spatial retransmission can boost streaming performance in diverse wireless environments. The benefits are most pronounced for busy networks, under fading conditions, or for mobile users.

Linear filters are ubiquitously used as a preprocessing step for many classification tasks in computer vision. In particular, applying Gabor filters followed by a classification stage, such as a support vector machine (SVM), is now common practice in computer vision applications like face identity and expression recognition. A fundamental problem occurs, however, with respect to the high dimensionality of the concatenated Gabor filter responses in terms of memory requirements and computational efficiency during training and testing. In this paper, we demonstrate how the preprocessing step of applying a bank of linear filters can be reinterpreted as manipulating the type of margin being maximized within the linear SVM. This new interpretation leads to sizable memory and computational advantages with respect to existing approaches. The reinterpreted formulation turns out to be independent of the number of filters, thereby allowing the examination of the feature spaces derived from arbitrarily large number of linear filters, a hitherto untestable prospect. Further, this new interpretation of filter banks gives new insights, other than the often cited biological motivations, into why the preprocessing of images with filter banks, like Gabor filters, improves classification performance.

We propose a novel method for removing irrelevant frames from a video given user-provided frame-level labeling for a very small number of frames. We first hypothesize a number of windows which possibly contain the object of interest, and then determine which window(s) truly contain the object of interest. Our method enjoys several favorable properties. First, compared to approaches where a single descriptor is used to describe a whole frame, each window's feature descriptor has the chance of genuinely describing the object of interest; hence it is less affected by background clutter. Second, by considering the temporal continuity of a video instead of treating frames as independent, we can hypothesize the location of the windows more accurately. Third, by infusing prior knowledge into the patch-level model, we can precisely follow the trajectory of the object of interest. This allows us to largely reduce the number of windows and hence reduce the chance of overfitting the data during learning. We demonstrate the effectiveness of the method by comparing it to several other semi-supervised learning approaches on challenging video clips.

Real-time stereo vision is attractive in many applications like robot navigation and 3-D scene reconstruction. Data parallel platforms, e.g., graphics processing unit (GPU), are often used for real-time stereo, because most stereo algorithms involve a large portion of data parallel computations. In this paper, we propose a stereo system on GPU which pushes the Pareto-efficiency frontline in the accuracy and speed tradeoff space. Our system is based on a hardware-aware algorithm design approach. The system consists of new algorithms and code optimization techniques. We emphasize on keeping the highly data parallel structure in the algorithm design process such that the algorithms can be effectively mapped to massively data parallel platforms. We propose two stereo algorithms: namely, exponential step size adaptive weight (ESAW), and exponential step size message propagation (ESMP). ESAW reduces computational complexity without sacrificing disparity accuracy. ESMP is an extension of ESAW, which incorporates the smoothness term to better model non-frontal planes. ESMP offers additional choice in the accuracy and speed tradeoff space. We adopt code optimization methodologies from the performance tuning community, and apply them to this specific application. Such an approach gives higher performance than optimizing the code in an “ad hoc” manner, and helps understanding the code efficiency. Experiment results demonstrate a speedup factor of 2.7-8.5 over state-of-the-art stereo systems at comparable disparity accuracy.

In many social settings, images of groups of people are captured. The structure of this group provides meaningful context for reasoning about individuals in the group, and about the structure of the scene as a whole. For example, men are more likely to stand on the edge of an image than women. Instead of treating each face independently from all others, we introduce contextual features that encapsulate the group structure locally (for each person in the group) and globally (the overall structure of the group). This “social context” allows us to accomplish a variety of tasks, such as such as demographic recognition, calculating scene and camera parameters, and even event recognition. We perform human studies to show this context aids recognition of demographic information in images of strangers.

The visual world demonstrates organized spatial patterns, among objects or regions in a scene, object-parts in an object, and low-level features in object-parts. These classes of spatial structures are inherently hierarchical in nature. Although seemingly quite different these spatial patterns are simply manifestations of different levels in a hierarchy. In this work, we present a unified approach to unsupervised learning of hierarchical spatial structures from a collection of images. Ours is a hierarchical rule-based model capturing spatial patterns, where each rule is represented by a star-graph. We propose an unsupervised EM-style algorithm to learn our model from a collection of images. We show that the inference problem of determining the set of learnt rules instantiated in an image is equivalent to finding the minimum-cost Steiner tree in a directed acyclic graph. We evaluate our approach on a diverse set of data sets of object categories, natural outdoor scenes and images from complex street scenes with multiple objects.

Higher order spatial features, such as doublets or triplets have been used to incorporate spatial information into the bag-of-local-features model. Due to computational limits, researchers have only been using features up to the 3rd order, i.e., triplets, since the number of features increases exponentially with the order. We propose an algorithm for identifying high-order spatial features efficiently. The algorithm directly evaluates the inner product of the feature vectors from two images to be compared, identifying all high-order features automatically. The algorithm hence serves as a kernel for any kernel-based learning algorithms. The algorithm is based on the idea that if a high-order spatial feature co-occurs in both images, the occurrence of the feature in one image would be a translation from the occurrence of the same feature in the other image. This enables us to compute the kernel in time that is linear to the number of local features in an image (same as the bag of local features approach), regardless of the order. Therefore, our algorithm does not limit the upper bound of the order as in previous work. The experiment results on the object categorization task show that high order features can be calculated efficiently and provide significant improvement in object categorization performance.

In this paper, we adopt constrained relaxation for distributed multi-view video coding (DMVC). The novel framework integrates the graph-based segmentation and matching to generate inter-view correlated side information without knowing the camera parameters. Moreover, graph-based representations of multi-view images are incorporated to form more distinctive feature constraints. The sparse data as a good hypothesis space aim for a best matching optimization of inter-view side information with compact syndromes, from inferred relaxed coset. The plausible filling-in from a priori feature constraints between neighboring views could reinforce a promising compensation to inter-view side information generation for joint multi-view decoding. In order to find distinctive feature matching with a more stable approximation, PCA-SIFT and TPS (thin plate spline) are adopted to reduce the dimension of SIFT descriptors and construct a more accurate inter-view motion model. The experimental results validate the high estimation precision and the rate-distortion improvements.

This paper addresses the performance optimization for scalable video coding and multicast over networks. Multi-path video streaming, network coding based routing, and network flow control are jointly optimized to maximize a network utility function defined over heterogeneous receivers. Importantly, contextual priors of scalable video layers are imposed on the flow routing optimization problem, seeking to guarantee the transmission cost for each layer in an incremental order and find jointly optimal multicast paths and associated rates. Through a primal decomposition and the primal-dual approach, a decentralized algorithm with two-level optimization update is developed to solve the target convex optimization problem. Numerical and simulation results validate the convergence and network performance of the proposed algorithm.

One important problem in computer vision is to provide a demographic description a person from an image. In practice, many of the state-of-the-art methods use only an analysis of the face to estimate the age and gender of a person of interest. We present a model that combines two problems, height estimation and demographic classification, which allows each to serve as context for the other. Our idea is to use a calibrated camera for measuring the height of people in the scene. Height is measured by jointly inferring across anthropometric dimensions, age, and gender using publicly available statistics. The height estimate provides context for recognizing the age and gender of the subject, and likewise age and gender conditions the distribution of the anthropometric features for estimating height. The performance of our method is explored on a new database of 127 people captured with a calibrated camera with recorded height, age, and gender. We show that estimating height leads to improvements in age and gender classification, and vice versa. To the best of our knowledge, our model produces the most accurate automatic height estimates reported, with the error having a standard deviation of 26.7 mm.

Interactive image segmentation is a powerful paradigm that allows users to direct the segmentation algorithm towards a desired output. However, marking scribbles on multiple images is a cumbersome process. Recent works show that statistics collected from user input in a single image can be shared among a group of related images to perform interactive cosegmentation. Most works use a naive heuristic of requesting the user input on a random image from the group. We show that in practice, selecting the right image to scribble on is critical to the resulting segmentation quality. In this paper, we address the problem of seed image selection, i.e., deciding which image among a group of related images should be presented to the user for scribbling. We formulate our approach as a classification problem and show that our approach outperforms the naive heuristic used by other works.

Relevance feedback has been taken as an essential tool to enhance content-based information retrieval systems by keeping the user in the retrieval loop. Among the fundamental relevance feedback approaches, feature space warping has been proposed as an effective approach for bridging the gap between high-level semantics and the low-level features. By examining the fundamental behavior of the feature space warping, we propose a new approach to harness its strength and resolve its weakness under various data distributions. Experiments on both synthetic data and real data reveal significant improvement from the proposed method.

This paper proposes a motion-focusing method to extract key frames and generate summarization synchronously for surveillance videos. Within each pre-segmented video shot, the proposed method focuses on one constant-speed motion and aligns the video frames by fixing this focused motion into a static situation. According to the relative motion theory, the other objects in the video are moving relatively to the selected kind of motion. This method finally generates a summary image containing all moving objects and embedded with spatial and motional information, together with key frames to provide details corresponding to the regions of interest in the summary image. We apply this method to the lane surveillance system and the results provide us a new way to understand the video efficiently.

Auto parking techniques are attracting more attention these days. In this paper, we develop an image-based method to estimate the depth contour in parking areas. Our algorithm is an extension of the canonical appearance-based models for object recognition. One challenge in object recognition is that limited training dataset can hardly represent all kinds intra-class and inter-class variations. We propose to augment the limited training dataset by on-the-spot learning from test data. The information is obtained by applying a fast block based stereo algorithm to estimate a rough disparity map. New “soft” samples are created to augment the training sample library. We present improved classification performance by using the proposed technique.

In this paper, we propose a local cost aggregation approach for real time stereo vision on a graphics processing unit (GPU). Recent research shows that local approaches based on carefully designed cost aggregation strategies can outperform many global approaches. Among those local aggregation approaches, adaptive-weight window produces the best quality disparity map under real-time constraint, but it is slower than other local approaches. We propose a very fast adaptive-weight aggregation method based on exponential step information propagation. The basic idea is to propagate information from long distance pixels within a few iterations. We also discuss important techniques of efficient implementation on GPU platform, which result in 10.5x speed up than a straightforward implementation. Compared to existing real time adaptive-weight approach, our technique reduces the computation time by more than half at improved accuracy. Detailed experimental results show that our technique is Pareto-optimal among existing real time or near real time stereo algorithms in the accuracy-speed trade-off space.

People are among the most popular subjects in photography, and in many social settings, images of groups of people are captured. People often arrange themselves in a very structured manner in these group images. For example, taller people might stand in a row behind smaller people. This structure is often exploited in captions that sequentially label the individuals in each row. We present an algorithm for automatically finding rows of people in group images. A graph is defined for the image, where each face is a vertex. Energy terms are learned from a training set of images. A minimum cut on this graph defines the rows of people in the image. On our test set, the algorithm achieves perfect results with 67.5% of the images. Detecting rows of people is useful for a number of applications.

Real-time streaming of audiovisual content over wireless networks is emerging as an important technology in multimedia communication. In this paper, we present a classification-based method to select appropriate relays in support of our prior work on Time-aware Opportunistic Relaying for video streaming over wireless mesh networks. The proposed scheme works as follows: each candidate relay runs a Support Vector Machine (SVM) to decide whether it should participate in forwarding packets for a source-destination pair or not. The SVM considers features that are sensitive to video quality and is trained with an aim to maximize user perceived quality. Our evaluation results show that the classificationbased approach outperforms the baseline scheme.

Forward error correction (FEC) coding and automatic repeat request (ARQ) are two commonly used techniques to tackle packet erasures in wireless video streaming. To preserve flexibility while reducing end-to-end latency, hybrid ARQ (HARQ) has been proposed as an alternative that combines the advantages of FEC and ARQ. This paper presents cooperative hybrid ARQ (CHARQ) that can further boost wireless video streaming quality. CHARQ takes advantage of the fact that, in the wireless environment, broadcast is free (from the sender's point of view) and that errors are location dependent so an intermediate proxy that overhears the source's transmission may transmit on behalf of the source to increase throughput efficiency. Simulation results show improved performance with the aid of cooperative proxies.

This paper presents an efficient opportunistic retransmission protocol (PRO, Protocol for Retransmitting Opportunistically) to improve the performance of IEEE 802.11 WLANs. PRO is a link-layer protocol that allows overhearing nodes to function as relays that retransmit on behalf of the source after they learn about a failed transmission. Relays with stronger connectivity to the destination have a higher chance of delivering the packet than the source does, thereby resulting in a more efficient use of the channel. PRO has four main features. First, channel reciprocity coupled with a run-time calibration process is used to estimate the instantaneous link quality to the destination. Second, a local qualification process filters out poor relays early. Third, a distributed relay selection algorithm chooses the best set of eligible relays among all qualified relays and prioritizes them. Finally, 802.11e Enhanced Distributed Channel Access (EDCA) is leveraged to make sure high priority relays transmit with high probability. PRO is designed to coexist with legacy 802.11 stations. We have implemented PRO in the driver of a commodity wireless card. Our extensive evaluation on both a controlled testbed and in the real world shows that PRO boosts throughput in diverse wireless environments, and especially in when there is significant contention for the channels, under fading, and with user mobility.

Pain is typically assessed by patient self-report. Self-reported pain, however, is difficult to interpret and may be impaired or in some circumstances (i.e., young children and the severely ill) not even possible. To circumvent these problems behavioral scientists have identified reliable and valid facial indicators of pain. Hitherto, these methods have required manual measurement by highly skilled human observers. In this paper we explore an approach for automatically recognizing acute pain without the need for human observers. Specifically, our study was restricted to automatically detecting pain in adult patients with rotator cuff injuries. The system employed video input of the patients as they moved their affected and unaffected shoulder. Two types of ground truth were considered. Sequence-level ground truth consisted of Likert-type ratings by skilled observers. Frame-level ground truth was calculated from presence/absence and intensity of facial actions previously associated with pain. Active appearance models (AAM) were used to decouple shape and appearance in the digitized face images. Support vector machines (SVM) were compared for several representations from the AAM and of ground truth of varying granularity. We explored two questions pertinent to the construction, design and development of automatic pain detection systems. First, at what level (i.e., sequence- or frame-level) should datasets be labeled in order to obtain satisfactory automatic pain detection performance? Second, how important is it, at both levels of labeling, that we non-rigidly register the face?

With the incoming of information era, web-based service has been developed rapidly and offered more and more business. These “open”, and widely “web enabled” applications are subject to greater and greater levels and types of attacks as hackers exploit vulnerabilities within the software like SQL Injection and Cross Site Scripts (XSS) attack. In this paper, we proposed a type of novel Embedded Markov Model (EMM) to detect different web application attacks, monitor the on-line user behavior and defend the malevolent user promptly. Comparing to previous web application attacks detecting approaches, our EMM approach can not only detect user’s invalidated input errors but also find out the unreasonable page transition behavior. By detecting unreasonable page transition, we can immediately defend the malevolent or silly user behavior to avoid the further web system failures and sensitive information disclosure. Furthermore, we implement an on-line user behavior surveillance system and use the real web traffic to evaluate the performance of our system. The experiment results show that our proposed EMM method can discover the abnormal behavior of malevolent user and detect the invalidated input attacks like SQL injection, XSS and string buffer overflow attacks.

10.5244/C.22.90

Variation due to viewpoint is one of the key challenges that stand in the way of a complete solution to the face recognition problem. It is easy to note that local regions of the face change differently in appearance as the viewpoint varies. Recently, patch-based approaches, such as those of Kanade and Yamada, have taken advantage of this effect resulting in improved viewpoint invariant face recognition. In this paper we propose a data-driven extension to their approach, in which we not only model how a face patch varies in appearance, but also how it deforms spatially as the viewpoint varies. We propose a novel alignment strategy which we refer to as “stack flow” that discovers viewpoint induced spatial deformities undergone by a face at the patch level. One can then view the spatial deformation of a patch as the correspondence of that patch between two viewpoints. We present improved identification and verification results to demonstrate the utility of our technique.

Spectral clustering and eigenvector-based methods have become increasingly popular in segmentation and recognition. Although the choice of the pairwise similarity metric (or affinities) greatly influences the quality of the results, this choice is typically specified outside the learning framework. In this paper, we present an algorithm to learn class-specific similarity functions. Mapping our problem in a Conditional Random Fields (CRF) framework enables us to pose the task of learning affinities as parameter learning in undirected graphical models. There are two significant advances over previous work. First, we learn the affinity between a pair of data-points as a function of a pairwise feature and (in contrast with previous approaches) the classes to which these two data-points were mapped, allowing us to work with a richer class of affinities. Second, our formulation provides a principled probabilistic interpretation for learning all of the parameters that define these affinities. Using ground truth segmentations and labellings for training, we learn the parameters with the greatest discriminative power (in an MLE sense) on the training data. We demonstrate the power of this learning algorithm in the setting of joint segmentation and recognition of object classes. Specifically, even with very simple appearance features, the proposed method achieves state-of-the-art performance on standard datasets.

Reseachers have verified that clothing provides information about the identity of the individual. To extract features from the clothing, the clothing region first must be localized or segmented in the image. At the same time, given multiple images of the same person wearing the same clothing, we expect to improve the effectiveness of clothing segmentation. Therefore, the identity recognition and clothing segmentation problems are inter-twined; a good solution for one aides in the solution for the other.

Recognizing people in images is one of the foremost challenges in computer vision. It is important to remember that consumer photography has a highly social aspect. The photographer captures images not in a random fashion, but rather to remember or document meaningful events in her life. The culture of the society of which the photographer is a part provides a strong context for recognizing the content of the captured images.

In computer vision, the bag-of-visual words image representation has been shown to yield good results. Recent work has shown that modeling the spatial relationship between visual words further improves performance. Previous work extracts higher-order spatial features exhaustively. However, these spatial features are expensive to compute. We propose a novel method that simultaneously performs feature selection and feature extraction. Higher-order spatial features are progressively extracted based on selected lower order ones, thereby avoiding exhaustive computation. The method can be based on any additive feature selection algorithm such as boosting. Experimental results show that the method is computationally much more efficient than previous approaches, without sacrificing accuracy.

Traditionally, object recognition is performed based solely on the appearance of the object. However, relevant information also exists in the scene surrounding the object. As supported by our human studies, this contextual information is necessary for accurate recognition in low resolution images. This scenario with impoverished appearance information, as opposed to using images of higher resolution, provides an appropriate venue for studying the role of context in recognition. In this paper, we explore the role of context for dense scene labeling in small images. Given a segmentation of an image, our algorithm assigns each segment to an object category based on the segment’s appearance and contextual information. We explicitly model context between object categories through the use of relative location and relative scale, in addition to co-occurrence. We perform recognition tests on low and high resolution images, which vary significantly in the amount of appearance information present, using just the object appearance information, the combination of appearance and context, as well as just context without object appearance information (blind recognition). We also perform these tests in human studies and analyze our findings to reveal interesting patterns. With the use of our context model, our algorithm achieves state-of-the-art performance on MSRC and Corel. datasets.

In this paper, a 3-layer Bayesian hierarchical detection framework (BHDF) is proposed for robust parking space detection. In practice, the challenges of the parking space detection problem come from luminance variations, inter- occlusions among cars, and occlusions caused by environmental obstacles. Instead of determining the status of parking spaces one by one, the proposed BHDF framework models the inter-occluded patterns as semantic knowledge and couple local classifiers with adjacency constraints to determine the status of parking spaces in a row-by-row manner. By applying the BHDF to the parking space detection problem, the available parking spaces and the labeling of parked cars can be achieved in a robust and efficient manner. Furthermore, this BHDF framework is generic enough to be used for various kinds of detection and segmentation applications.

Several classification scenarios employ multiple independently trained classifiers and the outputs of these classifiers need to be combined. However, since each of the trained classifiers exhibit different statistical characteristics, it is not appropriate to combine them using techniques that are blind to these differences. We propose a transform, dtransform, that transforms outputs of classifiers to approximate posterior probabilities, and caters to the statistical behavior of the classifier while doing so. The transformed outputs are now comparable, and can be combined using any of the classical combination rules. We show convincing results that demonstrate the effectiveness of the proposed transform in providing better estimates of the posterior probabilities as compared to standard transformations, as demonstrated by lower KL distance from the true distribution, higher classification accuracies and higher effectiveness of the standard classifier combination rules.

We propose an iterative method for discovering objects in images. In each iteration, the current estimate of the layout is processed by a sequence of perceptual grouping rules. Perceptual grouping appears to be the basis of visual organization of human. It is concerned with the problem of the formation of wholes from parts. The method does not rely on the mixture of Gaussian model and hence avoids the model selection problem. We use synthetic and real images to demonstrate that the obtained result is better than that obtained by other methods.

10.1145/1459359.1459458

Statistical pattern recognition techniques have recently been shown to provide a finer balance between misdetections and false alarms than the more conventional intrusion detection approaches, namely misuse detection and anomaly detection. A variety of classical machine learning and pattern recognition algorithms has been applied to intrusion detection with varying levels of success. We make two observations about intrusion detection. One is that intrusion detection is significantly more effective by using multiple sources of information in an intelligent way, which is precisely what human experts rely on. Second, different errors in intrusion detection have different costs associated with them-a simplified example being that a false alarm may be more expensive than a misdetection and, hence, the true objective function to be minimized is the cost of errors and not the error rate itself. We present a pattern recognition approach that addresses both of these issues. It utilizes an ensemble of a classifiers approach to intelligently combine information from multiple sources and is explicitly tuned toward minimizing the cost of the errors as opposed to the error rate itself. The information fusion approach dLEARNIN alone is shown to achieve state-of-the-art performances better than those reported in the literature so far, and the cost minimization strategy dCMS further reduces the cost with a significant margin.

We present a new approach to face relighting by jointly estimating the pose, reflectance functions, and lighting from as few as one image of a face. Upon such estimation, we can synthesize the face image under any prescribed new lighting condition. In contrast to commonly used face shape models or shape-dependent models, we neither recover nor assume the 3-D face shape during the estimation process. Instead, we train a pose- and pixel-dependent subspace model of the reflectance function using a face database that contains samples of pose and illumination for a large number of individuals (e.g., the CMU PIE database and the Yale database). Using this subspace model, we can estimate the pose, the reflectance functions, and the lighting condition of any given face image. Our approach lends itself to practical applications thanks to many desirable properties, including the preservation of the non-Lambertian skin reflectance properties and facial hair, as well as reproduction of various shadows on the face. Extensive experiments show that, compared to recent representative face relighting techniques, our method successfully produces better results, in terms of subjective and objective quality, without reconstructing a 3-D shape.

This paper presents a probabilistic framework for discovering objects in video. The video can switch between different shots, the unknown objects can leave or enter the scene at multiple times, and the background can be cluttered. The framework consists of an appearance model and a motion model. The appearance model exploits the consistency of object parts in appearance across frames. We use maximally stable extremal regions as observations in the model and hence provide robustness to object variations in scale, lighting and viewpoint. The appearance model provides location and scale estimates of the unknown objects through a compact probabilistic representation. The compact representation contains knowledge of the scene at the object level, thus allowing us to augment it with motion information using a motion model. This framework can be applied to a wide range of different videos and object types, and provides a basis for higher level video content analysis tasks. We present applications of video object discovery to video content analysis problems such as video segmentation and threading, and demonstrate superior performance to methods that exploit global image statistics and frequent itemset data mining techniques.

Summary form only given. Traditionally, signal processing is considered simply lowlevel processing. In the past decade, however, signal processing has grown to become the area where a variety of tools are created to solve high-level problems that conventionally would be studied by computer vision or machine learning researchers exclusively. For example, multiresolution analysis created popular image features like SIFT (scale-invariant feature transform), and statistical analysis gave birth to graphical models such as HMM (hidden Markov models) and topic models. In this talk, we will use one application to illustrate this growth of signal processing: object discovery, i.e., extracting the “object of interest” from a set of images in a completely unsupervised manner.

A calibration algorithm of two cameras using observations of a moving person is presented. Similar methods have been proposed for self-calibration with a single camera, but internal parameter estimation is only limited to the focal length. Recently it has been demonstrated that principal point supposed in the center of the image causes inaccuracy of all estimated parameters. Our method exploits two cameras, using image points of head and foot locations of a moving person, to determine for both cameras the focal length and the principal point. Moreover with the increasing number of cameras there is a demand of procedures to determine their relative placements. In this paper we also describe a method to find the relative position and orientation of two cameras: the rotation matrix and the translation vector which describe the rigid motion between the coordinate frames fixed in two cameras. Results in synthetic and real scenes are presented to evaluate the performance of the proposed method.

While face recognition techniques have rapidly advanced in the last few years, most of the work is in the domain of security applications. For consumer imaging applications, person recognition is an important tool that is useful for searching and retrieving images from a personal image collection. It has been shown that when recognizing a single person in an image, a maximum likelihood classifier requires the prior probability for each candidate individual. In this paper, we extend this idea and describe the benefits of using a group prior for identifying people in consumer images with multiple people. The group prior describes the probability of a group of individuals appearing together in an image. In our application, we have a subset of ambiguously labeled images for a consumer image collection, where we seek to identify all of the people in the collection. We describe a simple algorithm for resolving the ambiguous labels. We show that despite errors in resolving ambiguous labels, useful classifiers can be trained with the resolved labels. Recognition performance is further improved with a group prior learned from the ambiguous labels. In summary, by modeling the relationships between the people with the group prior, we improve classification performance.

The bag-of-words representation has attracted a lot of attention recently in the field of object recognition. Based on the bag-of-words representation, topic models such as probabilistic latent semantic analysis (PLSA) have been applied to unsupervised object discovery in still images. In this paper, we extend topic models from still images to motion videos with the integration of a temporal model. We propose a novel spatial-temporal framework that uses topic models for appearance modeling, and the probabilistic data association (PDA) filter for motion modeling. The spatial and temporal models are tightly integrated so that motion ambiguities can be resolved by appearance, and appearance ambiguities can be resolved by motion. We show promising results that cannot be achieved by appearance or motion modeling alone.

A successful representation of objects in the literature is as a collection of patches, or parts, with a certain appearance and position. The relative locations of the different parts of an object are constrained by the geometry of the object. Going beyond the patches on a single object, consider a collection of images of a particular class of scenes containing multiple (recurring) objects. The parts belonging to different objects are not constrained by such a geometry. However the objects, arguably due to their semantic relationships, themselves demonstrate a pattern in their relative locations, which also propagates to their parts. Analyzing the interactions between the parts across the collection of images would reflect these patterns, and the parts can be grouped accordingly. These groupings are typically hierarchical. We introduce hSO: Hierarchical Semantics of Objects, which is learnt from a collection of images of a particular scene and captures this hierarchical grouping. We propose an approach for the unsupervised learning of the hSO. The hSO simply holds objects, as clusters of patches, at its nodes, but it goes much beyond that and also captures interactions between the objects through its structure. In addition to providing the semantic layout of the scene, learnt hSOs can have several useful applications such as providing context for enhanced object detection and compact scene representation for scene category classification.

The Carnegie Mellon In Silico Vox project seeks to move best-quality speech recognition technology from its current software-only form into a range of efficient all-hardware implementations. The central thesis is that, like graphics chips, the application is simply too performance hungry, and too power sensitive, to stay as a large software application. As a first step in this direction, we describe the design and implementation of a fully functional speech-to-text recognizer on a single Xilinx XUP platform. The design recognizes a 1000 word vocabulary, is speaker-independent, recognizes continuous (connected) speech, and is a "live mode" engine, wherein recognition can start as soon as speech input appears. To the best of our knowledge, this is the most complex recognizer architecture ever fully committed to a hardware-only form. The implementation is extraordinarily small, and achieves the same accuracy as state-of-the-art software recognizers, while running at a fraction of the clock speed.

In this paper, we introduce profile view (PV) lip reading, a scheme for speaker-dependent isolated word speech recognition. We provide historic motivation for PV from the importance of profile images in facial animation for lip reading, and we present feature extraction schemes for PV as well as for the traditional frontal view (FV) approach. We compare lip reading results for PV and FV, which demonstrate a significant improvement for PV over FV. We show improvement in speech recognition with the integration of audio and visual features. We also found it advantageous to process the visual features over a longer duration than the duration marked by the endpoints of the speech utterance.

Topic models from the text understanding literature have shown promising results in unsupervised image categorization and object localization. Categories are treated as topics, and words are formed by vector quantizing local descriptors of image patches. Limitations of topic models include their weakness in localizing objects, and the requirement of a fairly large proportion of words coming from the object. We present a new approach that employs correspondences between images to provide information about object configuration, which in turn enhances the reliability of object localization and categorization. This approach is efficient, as it requires only a small number of correspondences. We show improved categorization and localization performance on real and synthetic data. Moreover, we can push the limits of topic models when the proportion of words coming from the object is very low.

We introduce hSOs: hierarchical semantics of objects. An hSO is learnt from a collection of images taken from a particular scene category. The hSO captures the interactions between the objects that tend to co-occur in the scene, and hence are potentially semantically related. Such relationships are typically hierarchical. For example, in a collection of images taken in a living room scene, the TV, DVD player and coffee-table co-occur frequently. The TV and the DVD player are more closely related to each other than the coffee table, and this can be learnt from the fact that the two are located at similar relative locations across images, while the coffee table is somewhat arbitrarily placed. The goal of this paper is to learn this hierarchy that characterizes the scene. The proposed approach, being entirely unsupervised, can detect the parts of the images that belong to the foreground objects, cluster these parts to represent objects, and provide an understanding of the scene by hierarchically clustering these objects in a semantically meaningful way - all from a collection of unlabeled images of a particular scene category. In addition to providing the semantic layout of the scene, learnt hSOs can have several useful applications such as compact scene representation for scene category classification and providing context for enhanced object detection.

Markov networks are an effective tool for the difficult but important problem of recognizing people in consumer image collections. Given a small set of labeled faces, we seek to recognize the other faces in an image collection. The constraints of the problem are exploited when forming the Markov network edge potentials. Inference is also used to suggest faces for the user to label, minimizing the work on the part of the user. In one test set containing 4 individuals, an 86% recognition rate is achieved with only 3 labeled examples.

We propose a novel learning algorithm to detect moving pedestrians from a stationary camera in real-time. The algorithm learns a discriminative model based on eigenflow, i.e., the eigenvectors derived from applying principal component analysis to the optical flow of moving objects, to differentiate between human motion patterns from other kind of motions like of cars etc. The learned model is a cascade of Adaboost classifiers of increasing complexity, with eigenflow vectors as the weak classifiers. Unlike some recent attempts to use motion for pedestrian detection, this system works in real-time. Moreover, the system is robust to small camera motion and slow illumination changes, and can detect moving children even though the training data had only adult pedestrians.

We present a novel approach to background cutout for image editing. We show how background cutout can be achieved without any user labeling. This is in contrast to current methods, where the user needs to label each image separately. Our method uses automatic object discovery methods to provide location and scale estimates of the object of interest; these estimates then provide seeds for initializing color distributions of a segmentation algorithm. We show that our approach can achieve similar performance as traditional methods that require users to specify for each image a bounding box of the target object.

In this paper, we propose a probabilistic framework for reconstructing scene geometry utilizing prior knowledge of a class of scenes, for example, scenes captured by a camera mounted on a vehicle driving through city streets. In this framework, we assume the video camera is calibrated, i.e., the intrinsic and extrinsic parameters are known all the time. While we assume a single camera moving during capturing, the framework can be generalized to multiple cameras as well. Traditional approaches try to match the points, lines or patches in multiple images to reconstruct scene geometry. The proposed framework also takes advantage of each patch's appearance and location to infer its orientation using prior information based on statistical learning from training data. The prior hence enhances the geometry reconstruction performance. We show that prior-based 3D reconstruction outperforms traditional 3D reconstruction with both synthetic data and real data, especially in the textureless areas.

Robust video streaming over time-varying, error-prone wireless LANs (WLANs) poses many challenges. In this paper, we present a time-based opportunistic relay (TOR) scheme for high-quality video streaming over WLANs. The proposed scheme exploits path diversity in relaying packets with awareness of the time constraints of video data. Specifically, relay nodes continuously overhear and opportunistically forward packets to enhance end-to-end delivery rates. To relay a packet in a time-aware manner, a relay deadline is computed for each packet and used by relay nodes to determine whether they should relay or discard the packet. Simulation results show that TOR improves video quality by up to 9.76 dB.

A major problem in metropolitan areas is searching for parking spaces. In this paper, we propose a novel method for parking space detection. Given input video captured by a camera, we can distinguish the empty spaces from the occupied spaces by using an 8-class support vector machine (SVM) classifier with probabilistic outputs. Considering the inter-space correlation, the outputs of the SVM classifier are fused together using a Markov random field (MRF) framework. The result is much improved detection performance, even when there are significant occlusion and shadowing effects in the scene. Experimental results are given to show the robustness of the proposed approach.

Pain is typically assessed by patient self-report. Self-reported pain, however, is difficult to interpret and may be impaired or not even possible, as in young children or the severely ill. Behavioral scientists have identified reliable and valid facial indicators of pain. Until now they required manual measurement by highly skilled observers. We developed an approach that automatically recognizes acute pain. Adult patients with rotator cuff injury were video-recorded while a physiotherapist manipulated their affected and unaffected shoulder. Skilled observers rated pain expression from the video on a 5-point Likert-type scale. From these ratings, sequences were categorized as no-pain (rating of 0), pain (rating of 3, 4, or 5), and indeterminate (rating of 1 or 2). We explored machine learning approaches for pain-no pain classification. Active Appearance Models (AAM) were used to decouple shape and appearance parameters from the digitized face images. Support vector machines (SVM) were used with several representations from the AAM. Using a leave-one-out procedure, we achieved an equal error rate of 19% (hit rate = 81%) using canonical appearance and shape features. These findings suggest the feasibility of automatic pain detection from video.

We propose a novel framework for 3D reassembly, the task of assembling a solid object from its broken pieces. The primary challenge in this under-explored problem is to robustly establish compatibility between parts from one object. Feature-based techniques have shown success in domains such as 3D similarity search; unfortunately, the global features typically employed to quantify whole-object similarity are unsuitable for identifying part-level compatibility. Therefore, we propose the use of local features which, in conjunction with robust matching, have become popular for object recognition in 2D images. This paper demonstrates that an analogous framework can be successful for 3D reassembly. Automating part-level compatibility enables the construction of an interactive system for 3D reassembly, where the user can easily assemble a desired object from a large collection of pieces (many of which are irrelevant) by iteratively selecting compatible parts. We evaluate our approach on a simulated database of broken objects and show that it scales well in the presence of noise and extraneous pieces

This paper presents a novel method for synthesizing a novel view from two sets of differently focused images taken by an aperture camera array for a scene consisting of two approximately constant depths. The proposed method consists of two steps. The first step is a view interpolation to reconstruct an all-in-focus dense light field of the scene. The second step is to synthesize a novel view by a light-field rendering technique from the reconstructed dense light field. The view interpolation in the first step can be achieved simply by linear filters that are designed to shift different object regions separately, without region segmentation. The proposed method can effectively create a dense array of pin-hole cameras (i.e., all-in-focus images), so that the novel view can be synthesized with better quality.

In this paper, we propose to capture image-based rendering scenes using a novel approach called active rearranged capturing (ARC). Given the total number of available cameras, ARC moves them strategically on the camera plane in order to minimize the sum of squared rendering errors for a given set of light rays to be rendered. Assuming the scene changes slowly so that the optimized camera locations are valid in the next time instance, we formulate the problem as a recursive weighted vector quantization problem, which can be solved efficiently. The ARC approach is verified on both synthetic and real-world scenes. In particular, a large self-reconfigurable camera array is built to demonstrate ARC's performance on real-world scenes. The system renders virtual views at 5-10 frames/s depending on the scene complexity on a moderately equipped computer. Given the virtual view point, the cameras move on a set of rails to perform ARC and improve the rendering quality on the fly

Abstract Video streaming over time‐varying, error‐prone wireless LANs (WLANs) poses many challenges. One problem is that WLANs are designed without awareness of the characteristics of application data, which causes performance degradation in a noisy or congested environment. In this paper, we propose a time‐based adaptive retry (TAR) mechanism for MPEG‐like video streaming over 802.11 wireless networks. TAR dynamically determines whether to send or discard a packet based on itsretransmission deadlineinstead of adopting a static retry limit uniformly over all the packets. Our approach can adapt the retry limit for each individual packet, thus providing indirect unequal error protection over different types of video frames. Analytical and simulation results show that TAR significantly improves video quality and saves channel bandwidth. We also describe a preliminary software‐based implementation of TAR and use it to demonstrate the practicality of the proposed approach. Copyright © 2007 John Wiley & Sons, Ltd.

Researchers in psychology have well studied the impact of the pose of a face as perceived by humans, and concluded that the so-called 3/4 view, halfway between the front view and the profile view, is the easiest for face recognition by humans. For face recognition by machines, while much work has been done to create recognition algorithms that are robust to pose variation, little has been done in finding the most representative pose for recognition. In this paper, we use a number of algorithms to evaluate face recognition performance when various poses are used for training. The result, similar to findings in psychology that the 3/4 view is the best, is also justified by the discrimination power of different regions on the face, computed from both the appearance and the geometry of these regions. We believe our study is both scientifically interesting and practically beneficial for many applications.

Most pose robust face verification algorithms, which employ 2D appearance, rely heavily on statistics gathered from offline databases containing ample facial appearance variation across many views. Due to the high dimensionality of the face images being employed, the validity of the assumptions employed in obtaining these statistics are essential for good performance. In this paper we assess three common approaches in 2D appearance pose mismatched face recognition literature. In our experiments we demonstrate where these approaches work and fail. As a result of this analysis, we additionally propose a new algorithm that attempts to learn the statistical dependency between gallery patches (i.e. local regions of pixels) and the whole appearance of the probe image. We demonstrate improved performance over a number of leading 2D appearance face recognition algorithms.

In this paper, we propose a general object detection framework which combines the Hidden Markov Model with the Discriminative Random Fields. Recent object detection alrithms have achieved impressive results by using graphical models, such as Markov Random Field. These models, however, have only been applied to two dimensional images. In many scenarios, video is the directly available source rather than images, hence an important information for detecting objects has been omitted — the temporal information. To demonstrate the importance of temporal information, we apply graphical models to the task of text detection in video and compare the result of with and without temporal information. We also show the superiority of the proposed models over simple heuristics such as median filter over time.

We propose a novel unsupervised learning algorithm to extract the layout of an image by learning latent object-related aspects. Unlike traditional image segmentation algorithms that segment an image using feature similarity, our method is able to learn high-level object characteristics (aspects) from a large number of unlabeled images containing similar objects to facilitate image segmentation. Our method does not require human to annotate the training set and works without supervision. We use a graphical model to address the learning of aspects and layout extraction together. In particular, aspect-feature dependency from multiple images is learned via the expectation-maximization algorithm. We demonstrate that, by associating latent aspects to spatial structure, the proposed method achieves much better layout extraction results than using probabilistic latent semantic analysis.

We propose a novel video-based rendering algorithm with a single moving camera. We reconstruct a dynamic 3D model of the scene with a feature point set that "evolves" over time. As the scene's appearance changes due to camera and object motions, some existing feature points dynamically disappear while some new feature points dynamically appear relative to the camera. The newly generated feature points' 3D positions and motions are initialized using nearby existing feature points' positions and motions. Our feature evolution, when incorporated into standard tracking and 3D reconstruction algorithms, provides for robust and dense 3D meshes, and their corresponding motions. Consequently, the evolution-based, time-dependent 3D meshes, motions, and textures render good-quality images at a virtual viewpoint and at a desired time instance. We also extend the proposed video-based rendering algorithm from using one single moving camera with one reconstructed depth map to using multiple moving cameras with multiple reconstructed depth maps to avoid occlusion and improve the rendering quality.

Content-free image retrieval uses accumulated user feedback records to retrieve images without analyzing image pixels. We present a Bayesian-based algorithm to analyze user feedback and show that it outperforms a recent maximum entropy content-free algorithm, according to extensive experiments on trademark logo and 3D model datasets. The proposed algorithm also has the advantage of being applicable to both content-free and traditional content-based image retrieval, thus providing a common framework for these two paradigms

Location-aware computing enables automatic tailoring of information and services based on the current location of a mobile user. We have designed and implemented CMUseum, a system that enables location-based video streaming applications, as well as other add-on services on top of the purposed infrastructure. Our novel design incorporates an 802.15.4 Zigbee sensor network for collecting location information as well as an 802.11 Wi-Fi network for streaming video contents. This framework is largely event-driven in order to support the real-time nature of the video streaming services it handles. We have demonstrated the viability of this system through implementing a working system in a museum tour guide scenario

To achieve much faster decoding, or much lower power consumption, we need to liberate speech recognition from the artificial constraints of its current software-only form, and move the essential computations directly into silicon. There are vast efficiencies waiting to be unlocked in this application – we need the proper architecture to do so. We report results from a firstgeneration hardware architecture simulated at bit-level, and a complete, working FPGA-based prototype. Simulation results show that rather modest hardware designs, running 10-20X slower than conventional processors, can already decode at 0.6 xRT, running the standard 5K Wall Street Journal benchmark.

10.5244/C.19.73

Researchers have been working on human face recognition for decades. Face recognition is hard due to different types of variations in face images, such as pose, illumination and expression, among which pose variation is the hardest one to deal with. To improve face recognition under pose variation, this paper presents a geometry assisted probabilistic approach. We approximate a human head with a 3D ellipsoid model, so that any face image is a 2D projection of such a 3D ellipsoid at a certain pose. In this approach, both training and test images are back projected to the surface of the 3D ellipsoid, according to their estimated poses, to form the texture maps. Thus the recognition can be conducted by comparing the texture maps instead of the original images, as done in traditional face recognition. In addition, we represent the texture map as an array of local patches, which enables us to train a probabilistic model for comparing corresponding patches. By conducting experiments on the CMU PIE database, we show that the proposed algorithm provides better performance than the existing algorithms.

We propose a face mosaicing approach to model both the facial appearance and geometry from pose-varying videos, and apply it in face tracking and recognition. The basic idea is that by approximating the human head as a 3D ellipsoid, multi-view face images can be back projected onto the surface of the ellipsoid, and the surface texture map is decomposed into an array of local patches. During the online modeling process, the position and pose of the first frame is assumed to be known for a given video sequence. For each frame in the sequence, the algorithm estimates the face-position and pose, and generates a texture map, which is further utilized in updating the mosaic model.

For information retrieval, relevance feedback is an important technique. This paper proposes a relevance feedback technique which is based on a probabilistic framework. The binary feature vectors in our experiment are high-level semantic features of trademark logo images, each feature representing the presence or absence of a certain shape or object. The images were labeled by human experts of the trademark office. We compared our probabilistic method with several existing methods such as MARS, MindReader, and one-class SVM. Our method outperformed the others.

With image-based relighting (IBL), one can render realistic relit images of a scene without prior knowledge of object geometry in the scene. However, traditional IBL methods require a large number of basis images, each corresponding to a lighting pattern, to estimate the surface reflectance function (SRF) of the scene. We present a statistical approach to estimating the SRF which requires fewer basis images. We formulate the SRF estimation problem in a signal reconstruction framework. We use principal component analysis (PCA) (Duda, R.O. et al., "Pattern Classification, 2nd edition", p.115-17, Wiley Interscience, 2000) to show that the most effective lighting patterns for the data acquisition process are the SRF covariance matrix eigenvectors corresponding to the largest eigenvalues. In addition, we show that, for typical SRFs, especially when the objects have Lambertian surfaces, DCT-based lighting patterns perform as well as the optimal PCA-based lighting patterns. We compare SRF estimation performance of the statistical approach with traditional IBL techniques. Experimental results show that the statistical approach can achieve better performance with fewer basis images.

Image based rendering (IBR) is basically a light ray resampling method for generating a novel image without aliasing artifacts from a given set of sampled light rays. To avoid aliasing artifacts, conventional methods require an estimate of the scene geometry. In this paper, we present a novel IBR method by linear filtering without estimating the scene geometry, for the simplified case when generating a virtual image at the center of 2/spl times/2 sparse camera array for a two depth-layers scene. The reconstruction filter used in the proposed method can derive by integrating all the process in our previously proposed method into a one-shot process.

In this paper, we study biometric key generation using face images. Given a face image, a set of biometric features are extracted. Each feature is compared with a threshold to generate a key-bit. By cascading the key-bits from all the features, we obtain one bio-key that can be used for security applications. The performance of a biometric key generation system, determined by the chosen thresholds, can be evaluated according to reliability and security. A generalized optimal thresholding method is proposed in this paper to improve the reliability by minimizing the authentication error rate and the security by maximizing the entropy of the generated key.

We introduce a novel approach to capturing light field with lensless cameras. By moving the cameras back and forth, we capture a set of images. We show that it is possible to reconstruct the light field from these blurry images. The problem is formulized in a way similar to computer tomography, so that the light field can be reconstructed using existing algorithms. The light field can then be used to render 3D scenes. Synthetic examples are presented to show the effectiveness of the proposed method.

Robust video streaming over error-prone wireless LANs (WLANs) poses many challenges. In this paper, we propose a timestamp-based content-aware adaptive retry (CAR) mechanism for MPEG video streaming over 802.11 WLANs, where the MAC dynamically determines whether to send or discard a packet based on its retransmission deadline. The retransmission deadline is assigned to each packet according to its temporal relationship and error propagation characteristics with respect to other video packets in the same GOP. The proposed scheme avoids late packets by eliminating the impact of random backoff deference and co-channel interference with proper initial delay introduced at the receiver. Simulation results show CAR significantly improves video quality and saves channel bandwidth.

Summary form only given. The performance of a content-based information retrieval (CBIR) system is very subjective and hence user-dependent. To the user, similarity between objects in the database is often high-level and semantic. However, features extracted from objects directly in their digital representations are often low-level features. The gap between low-level features and high-level semantics has been the major obstacle to better retrieval performance. In this talk, we outline several approaches to bridging the gap between low-level features and high-level semantics, including hidden annotation and relevance feedback. We present a few specific techniques: active learning, annotation propagation, feature space warping, and semantic metric linking, all aiming at propagating the semantics from some objects to the others.

In this paper, an in-depth analysis is undertaken into effective strategies for integrating the audio-visual speech modalities with respect to two major questions. Firstly, at what level should integration occur? Secondly, given a level of integration how should this integration be implemented? Our work is based around the well-known hidden Markov model (HMM) classifier framework for modeling speech. A novel framework for modeling the mismatch between train and test observation sets is proposed, so as to provide effective classifier combination performance between the acoustic and visual HMM classifiers. From this framework, it can be shown that strategies for combining independent classifiers, such as the weighted product or sum rules, naturally emerge depending on the influence of the mismatch. Based on the assumption that poor performance in most audio-visual speech processing applications can be attributed to train/test mismatches we propose that the main impetus of practical audio-visual integration is to dampen the independent errors, resulting from the mismatch, rather than trying to model any bimodal speech dependencies. To this end a strategy is recommended, based on theory and empirical evidence, using a hybrid between the weighted product and weighted sum rules in the presence of varying acoustic noise for the task of text-dependent speaker recognition.

This paper presents a new approach for virtual view synthesis that does not require any information of scene geometry. Our approach first generates multiple virtual views at the same position based on multiple depths by the conventional view interpolation method. The interpolated views suffer from blurring and ghosting artifacts due to the pixel mis-correspondence. Secondly, the multiple views are integrated into a novel view where all regions are focused. This integration problem can be formulated as the problem of solving a set of linear equations that relates the multiple views. To solve this set of equations, two methods using projection onto convex sets (POCS) and inverse filtering are presented that effectively integrate the focused regions in each view into a novel view. Experimental results using real images show the validity of our methods.

This paper introduces a shape descriptor, the soft shape context, motivated by the shape context method. Unlike the original shape context method, where each image point was hard assigned into a single histogram bin, we instead allow each image point to contribute to multiple bins, hence more robust to distortions. The soft shape context can easily be integrated into the iterative closest point (ICP) method as an auxiliary feature vector, enriching the representation of an image point from spatial information only, to spatial and shape information. This yields a registration method more robust than the original ICP method. The method is general for 2D shapes. It does not calculate derivatives, hence being able to handle shapes with junctions and discontinuities. We present experimental results to demonstrate the robustness compared with the standard ICP method.

In this paper, we propose an algorithm for view-dependent nonuniform sampling for image-based rendering (IBR). Given a set of virtual views, the positions of the capturing cameras are rearranged in order to obtain the optimal rendering quality. The resulting arrangement of the cameras is effectively nonuniform sampling of the plenoptic function. We formulate the above sampling problem as a recursive weighted vector quantization problem, which can be solved efficiently. Experimental results show that the nonuniform sampling scheme renders much better images than traditional uniform sampling methods.

In this paper, we introduce a novel method for key generation based on biometrics. Given the biometrics, a set of reliable features are extracted. Each feature is compared with multiple thresholds to generate a multi-bit key. By cascading the multi-bit keys, we obtain one biokey that can be used for security applications. In order to generate a reliable bio-key, an optimal thresholding method is proposed to minimize the authentication error rate, in terms of the false accept rate (FAR) and the false reject rate (FRR). The experimental results show that the proposed approach of key generation is user-friendly and reliable.

In this paper, we analyze the security problem of user information associated key generation (UIAKG) systems. We consider three kinds of attacks from the hacker: the exhaustive search attack, the authentic key statistics attack and the device key statistics attack. Under each attack, we give the estimate of the number of guesses the hacker has to make in order to access the system. Such analysis provides useful guidelines in designing a UIAKG system. The analysis also suggests that a user-dependent UIAKG is more secure than a user-independent one. Two UIAKG systems using face images as inputs are designed and compared to support the above theoretical analysis.

Relevance feedback has been a very useful tool to enhance the performance of content-based information retrieval (CBIR) systems. To fully make use of the precious user feedback provided to a system, we propose an approach named semantic propagation, which reveals the deep semantic relationships among objects in the database, given a set of relevance feedbacks between object pairs. In particular, we present two semantic propagation algorithms that are applicable to CIBR systems with a feature vector space model and a general metric space model, respectively. Experiments on a 3D model retrieval system and a logo image retrieval system are performed to show the effectiveness of the proposed methods.

Instead of using PINs and passwords as cryptographic keys that are either easy to forget or vulnerable to dictionary attacks, easy-to-carry and difficult-to-transfer keys can be generated based on user-specific biometric information. A framework is proposed to generate stable cryptographic keys from biometric data that is unstable in nature. The proposed framework differs from prior work in that user-dependent transforms are utilized to generate more compact and distinguishable features. Thereby, a longer and more stable bitstream can be generated as the cryptographic key. Experiments are performed on a face database to verify the feasibility of the proposed framework. The preliminary result is very encouraging.

The work presents a novel method for synthesizing a novel view from two sets of differently focused images taken by a sparse camera array for a scene of two approximately constant depths. The proposed method consists of two steps. The first step is a view interpolation to reconstruct an all-focused dense light field of the scene. The second step is to synthesize a novel view by a light-field rendering technique from the reconstructed dense light field. The view interpolation can be achieved simply by linear filters that are designed to convert defocus effects to parallax effects without estimating the depth map of the scene. The proposed method can effectively create a dense array of pin-hole cameras (i.e., all-focused images), so that the final novel view is better than traditional method using a sparse array of cameras. Experimental results on real images from four aligned cameras are shown.

Recent convergence of image processing, computer vision, and computer graphics has resulted in an exciting research topic referred to as image-based rendering (IBR). Widely used in applications ranging from movie special effects (e.g., "the Matrix") to building virtual environments, IBR has become a critical tool for creating visually exciting content. With IBR, real-world scenes can be captured and rendered directly from images captured by cameras, eliminating the need for computationally expensive modeling of 3D geometry or surface reflectance, as is often done in traditional computer graphics. Various approaches to IBR have been proposed to render the scenes correctly and effectively given the captured images. We propose an active scene-capturing algorithm to efficiently capture the images. Based on the images that have been taken and the geometry information known so far, the algorithm intelligently determines where to pose the camera to capture the scene for best rendering performance. This results in a nonuniform but optimal set of captured images.

While traditional face recognition is typically based on still images, face recognition from video sequences has become popular. In this paper, we propose to use adaptive hidden Markov models (HMM) to perform video-based face recognition. During the training process, the statistics of training video sequences of each subject, and the temporal dynamics, are learned by an HMM. During the recognition process, the temporal characteristics of the test video sequence are analyzed over time by the HMM corresponding to each subject. The likelihood scores provided by the HMMs are compared, and the highest score provides the identity of the test video sequence. Furthermore, with unsupervised learning, each HMM is adapted with the test video sequence, which results in better modeling over time. Based on extensive experiments with various databases, we show that the proposed algorithm results in better performance than using majority voting of image-based recognition results.

We introduce a new classification algorithm based on the concept of symmetric maximized minimal distance in subspace (SMMS). Given the training data of authentic samples and imposter samples in the feature space, SMMS tries to identify a subspace in which all the authentic samples are close to each other and all the imposter samples are far away from the authentic samples. The optimality of the subspace is determined by maximizing the minimal distance between the authentic samples and the imposter samples in the subspace. We present a procedure to achieve such optimality and to identify the decision boundary. The verification procedure is simple since we only need to project the test sample to the subspace and compare it against the decision boundary. Using face authentication as an example, we show that the proposed algorithm outperforms several other algorithms based on support vector machines (SVM).

We introduce a static error analysis technique, based on smart interval methods from affine arithmetic, to help designers translate DSP codes from full-precision floating-point to smaller finite-precision formats. The technique gives results for numerical error estimation comparable to detailed simulation, but achieves speedups of three orders of magnitude by avoiding actual bit-level simulation. We show results for experiments mapping common DSP transform algorithms to implementations using small custom floating point formats.

In recent years, the importance of 2D maps and 3D models has grown significantly, especially due to the popularity of map data over the Web and computer graphics used in entertainment and e-commerce applications. We review the concept of image watermarking, and extend it to 2D/3D vector data. We present a number of watermarking algorithms for 2D electronic maps and 3D wire-frame models. We show that invisible watermarks can be added to 2D/3D data and detected reliably. The watermarks can be designed to be robust to quantization of the vector data and several other hostile attacks.

Streaming of precoded video, which is both source and channel coded, over wireless networks faces challenges of time-varying packet loss rate and fluctuating bandwidth. Rate shaping has been proposed (Chen, T.P.-C. and Chen, T., ICASSP, 2002) to "shape" the precoded video to adapt to the real-time bandwidth requirement and the packet loss rate. We now propose a novel "fine-grained rate shaping" (FGRS) scheme to allow for bandwidth adaptation over a wide range of bandwidth and packet loss rate. The video is precoded with fine granularity scalability (FGS) followed by forward error correction (FEC) coding with erasure codes. Utilizing the fine granularity property of FGS and FEC, FGRS selectively drops part of the precoded video and still yields a decodable bitstream at the decoder. A new two-stage rate-distortion (R-D) optimization, with model-based hyper-plane and hill-climbing based refinement, is proposed to select which part of the precoded video to drop. Promising results for FGRS are shown.

During the development of floating-point signal processing systems, an efficient error analysis method is needed to guarantee the output quality. We present a novel approach to floating-point error bound analysis based on affine arithmetic. The proposed method not only provides a tighter bound than the conventional approach, but also is applicable to any arithmetic operation. The error estimation accuracy is evaluated across several different applications which cover linear operations, nonlinear operations, and feedback systems. The accuracy decreases with the depth of computation path and also is affected by the linearity of the floating-point operations.

Sketching is a natural way of input that provides an effective means of illustration. A sketch consists of multiple strokes that can be captured by pen-based devices to be stored in a database for future retrieval. In our prior work, we proposed a sketch retrieval method that is suitable for whole matching. However, the problem gets complicated if the retrieval is performed for partially matched sketches. In this paper, we propose a method to solve the partial matching problem for retrieving sketches. In the matching stage, the score is first computed by considering the stroke feature similarity alone. For a sketch consisting of multiple strokes, spatial relations between every pair of strokes (bistroke) can be used together with the bistroke features for finding the correspondence to further improve the retrieval performance. A potential application of sketch retrieval for partial matching is to retrieve materials from the lecture notes for reviewing without the trouble of describing it using keywords.

We apply generalized sampling to image-based rendering (IBR) data, more specifically, the lightfield. We show that, in theory, the lowest sampling rate of a lightfield when we use generalized sampling can be as low as half of that when we use rectangular sampling. However, in practice, rectangular sampling has several advantages over generalized sampling. We analyze the pros and cons for each sampling approach, and explain why, in practice, rectangular sampling is still preferable.

In this paper, we introduce the surface plenoptic function (SPF) as a tool for the sampling analysis of image-based rendering (IBR). SPF is a plenoptic function defined on the object surface. It can be mapped onto the normal IBR representation via a coordinate transform. By assuming some properties of the SPF, we can provide insightful analysis on the sampling of the IBR representation, for both uniform and non-uniform methods.

The modeling of facial appearance has many applications. This paper proposes an approach to generating a statistical face model based on video mosaicing. Unlike traditional video mosaicing, we use the geometry of a face to improve the mosaicing result. Given a face sequence, each frame is unwrapped onto certain portion of the surface of a sphere, as determined by spherical projection and the minimization procedure using the Levenberg-Marquardt algorithm. A statistical model containing a mean image and a number of eigenimages, instead of only one image template, is used to represent the face mosaic. Good experimental results have been observed.

In this paper, we describe a retrieval system that uses hidden annotation to improve the performance. The contribution of this paper is a novel active learning framework that can improve the annotation efficiency. For each object in the database, we maintain a list of probabilities, each indicating the probability of this object having one of the attributes. This list of probabilities serves as the basis of our active learning algorithm, as well as semantic features to determine the similarity between objects in the database. We show active learning has better performance than random sampling in all our experiments.

Streaming of preceded video, which is both source- and channel-coded, over packet-loss networks faces challenges of the time-varying packet loss rate and fluctuating bandwidth. Rate shaping has been proposed to reduce the bit rate of a preceded video bitstream to adapt to the real-time bandwidth variation. In our earlier work, rate shaping was extended to consider not only the bandwidth but also the packet loss rate variations. In practice, the reconstructed result of the previous frame will affect the following frames if the video is predictive coded, and/or the error concealment method performed at the receiver utilizes temporal information. However, none prior work considers rate shaping for pre source- and channel-coded video with such frame dependency. In order to incorporate frame dependency into the rate shaping process, we propose to send the location (and mean) of the corrupted macroblock back to the sender, and use such feedback information to determine the distortion measure in the rate-distortion optimized rate shaping. Experiments have shown improved performance with the aids of the feedback information.

As pen-based devices become more popular, it has motivated several novel research issues in the area of pen computing. One interesting and practical problem is the storage and retrieval of hand-drawn sketches. A sketch can consist of handwritten notes, symbols, free-form hand-drawings, annotations on a document, and others. It is very useful to store the sketches in a database and then retrieve them later. In our prior work, we proposed a method for hand-drawn sketch retrieval by performing stroke-by-stroke matching and by considering the spatial relationship between them. In this paper, we propose a method to simplify a sketch and perform matching in a hierarchical manner. With this approach, a shaded region or a region of complex strokes can be detected automatically and represented as a single hyper-stroke. In the matching stage, the similarity is computed by combining the similarity scores across the feature spaces at each level and the similarity of the stroke hierarchies. A sketch may be better represented by exploiting the structural relationship between the strokes. In addition, as a general rule to the hierarchical approach, it has the advantage of reduced computation thus the retrieval process would speed up.

Rapid speaker adaptation is becoming more important in emerging applications where storage, computation and training utterances are at a premium (e.g. PDAs, cell phones). Effective adaptation can be achieved for the task of speaker verification, based on a maximum a posteriori (MAP) learning framework, by restricting the client's parametric model to be a linear combination of parameters estimated from training observations and a speaker independent "world" model (i.e. relevance adaptation (RA)). Subspace adaptation (SA) attempts to restrict a client's parametric representation to a pre-defined subspace during estimation. In this paper we elucidate where subspace adaptation outperforms world adaptation, demonstrate where and why subspace adaptation is sometimes not as effective and give insights into what cost criteria should be used to construct the adaptation parametric subspace. Results are presented on the acoustic portion of the XM2VTS database for the task of Gaussian mixture model (GMM) based text-independent speaker verification.

In this paper, we develop a system for active image-based rendering (IBR). Active IBR is a framework that is capable of estimating the final rendering quality and capturing the next view at the position where the rendering quality can be improved the most. The result is a non-uniform capturing scheme for IBR. Experimental results on synthetic scenes have shown that active IBR outperforms uniformly captured IBR. In this paper, we set up an active IBR system that can capture real objects with off-the-shelf components. We show that our system can capture objects more intelligently than uniform capturing.

We introduce an improved classification algorithm based on the concept of symmetric maximized minimal distance in subspace (SMMS). Given the training data of authentic samples and imposter samples in the feature space, our previous approach, SMMS, tried to identify a subspace in which all the authentic samples were projected onto the origin and all the imposter samples were far away from the origin. The optimality of the subspace was determined by maximizing the minimal distance between the origin and the imposter samples in the subspace. The generalized SMMS relaxes the constraint of fitting all the authentic samples to the origin in the subspace to achieve the optimality and considers the optimal direction of the linear support-vector machines (SVM) as a feasible solution in our optimization procedure to guarantee that our result is no worse than the linear SVM. We present a procedure to achieve such optimality and to identify the subspace and the decision boundary. Once the subspace is trained, the verification procedure is simple since we only need to project the test sample onto the subspace and compare it against the decision boundary. Using face authentication as an example, we show that the proposed algorithm outperforms the linear classifier based on SMMS and SVM. The proposed algorithm also applies to multimodal feature spaces. The features can come from any modalities, such as face images, voices, fingerprints, etc.

Image-based rendering (IBR) has become a very active research area in recent years. The spectral analysis problem for IBR has not been completely solved. In this paper, we present a new method to parameterize the problem, which is applicable for general-purpose IBR spectral analysis. We notice that any plenoptic function is generated by light ray emitted/reflected/refracted from the object surface. We introduce the surface plenoptic function (SPF), which represents the light rays starting from the object surface. Given that radiance along a light ray does not change unless the light ray is blocked, SPF reduces the dimension of the original plenoptic function to 6D. We are then able to map or transform the SPF to IBR representations captured along any camera trajectory. Assuming some properties on the SPF, we can analyze the properties of IBR for generic scenes such as scenes with Lambertian or non-Lambertian surfaces and scenes with or without occlusions, and for different sampling strategies such as lightfield/concentric mosaic. We find that in most cases, even though the SPF may be band-limited, the frequency spectrum of IBR is not band-limited. We show that non-Lambertian reflections, depth variations and occlusions can all broaden the spectrum, with the latter two being more significant. SPF is defined for scenes with known geometry. When the geometry is unknown, spectral analysis is still possible. We show that with the "truncating windows" analysis and some conclusions obtained with SPF, the spectrum expansion caused by non-Lambertian reflections and occlusions can be quantatively estimated, even when the scene geometry is not explicitly known. Given the spectrum of IBR, we also study how to sample IBR data more efficiently. Our analysis is based on the generalized periodic sampling theory with arbitrary geometry. We show that the sampling efficiency can be up to twice of that when we use rectangular sampling. The advantages and disadvantages of generalized periodic sampling for IBR are also discussed.

We present a client/server-based un-intrusive collaborative multimedia application - called eMeeting - that extracts and archives salient visual information from seminars, while allowing interaction among presenters and the audience in real time. The application uses APIs of the Lotus Sametime/spl reg/ system such as secure login, directory, meeting and multimedia services, to which we have added our own extensions. These extensions use the binary messaging channel of Sametime and allow us to add stroke-based chat as well as stroke-based annotations of slides. This permits communication among all clients. In parallel with the services just mentioned and in the same Java application, users of eMeeting are presented with slides that the system extracts from the presenter's visual material using an automatic recognition algorithm that classifies video frames as slides. At this time we assume that the system may or may not have access to the visual content in electronic form. Users can follow the seminar in the form of presenter's video, and as a series of extracted images. The server transmits the slides cyclically to each user using IP multicast giving each client the opportunity to select any previous slide. Thus, late joiners can browse slides that have been accumulated by the system.

We present in this paper an adaptive joint source-channel coding scheme using rate shaping on pre-coded video data. Rate shaping selectively drops portions of the video bitstream before transmitting them in order to satisfy the network bandwidth requirement. In wireless multimedia transport over heterogeneous networks, limited bandwidth is not the only issue. The high error rate of the channel should be considered as well, so channel coding is often applied. We propose a rate shaping method that drops not only the source-coding segments of the video bitstream, but also the channel-coding segments of the video bitstream, adaptively according to the network condition, in order to achieve the optimal rate-distortion performance. The proposed method is based on “discrete rate-distortion combination” to accomplish joint source-channel coding. We consider both the simulcast and multicast scenarios and show promising results.

To enable floating-point (FP) signal processing applications in low-power mobile devices, we propose a lightweight FP design flow that can optimize the bit-width configuration. The optimization considers both the hardware cost and the numerical precision. Variable grouping is used to reduce the complexity of optimization by connecting software description and hardware implementation. The optimization algorithm is able to avoid local optima, and multiple-phase optimization helps to reduce the cost further. We apply the proposed design flow to the design of inverse discrete cosine transform (IDCT), and show that the power consumption of our lightweight FP IDCT is comparable to an optimized fixed-point design. In addition, promising results on some real-world applications such as video coding and speech recognition demonstrate that lightweight FP signal processing will find more and more applications in low-power devices.

In this paper we propose a method to retrieve free-form hand-drawn sketches stored in the form of multiple strokes, by extracting the shape information for each stroke and by considering the geometric relationship between the strokes. To extract the shape information, a number of shape estimators are applied to each stroke to provide a soft decision about how similar it is to a particular shape type. Then, two strokes are matched according to a specific set of features for each shape type. The proximity of the corresponding strokes is used to account for the geometric relationship between multiple strokes during the matching stage. Our approach is robust to different drawing styles, thus making our retrieval system user-independent. Sketch retrieval is useful in applications where a user can easily search through a database of hand-draw sketches by inputting a sketch about what he/she is looking for, without the trouble of describing it using keywords.

In multimedia information retrieval, shot boundary detection is a very active research topic. In order to perform shot boundary detection, we propose an algorithm for modeling temporal statistics using a novel eigenspace updating method. The feature extracted from the current frame is compared with a model trained from features in the previous frames. A shot boundary is detected if the new feature does not fit well to the existing model. The model is based on principal component analysis (PCA), or the eigenspace method, in which the eigenspace can be updated to capture the non-stationary statistics of the features. The experiment results show that the proposed algorithm outperforms the traditional direct differencing method.

In this paper, we propose a new eigenfilter approach to designing least square error filters. The filters are obtained by finding an eigenvector of a real, symmetric and positive definite matrix, which is numerically stable. The proposed algorithm has two advantages. First, we show that the least-square solution, which can only be obtained through matrix inversion in the literature, can be asymptotically reached with our algorithm. Second, when numerical errors break. the matrix inversion method, our algorithm can still find some ìoptimalî filter through tuning an internal parameter.

Relevance feedback has been shown to be an effective tool to enhance content-based information retrieval (CBIR) systems. We propose a new approach to relevance feedback by warping the database's feature space, or shifting the objects' data points in a controlled manner responding to user feedback. We demonstrate that given consistent feedback, the performance of the retrieval system can be significantly increased.

Video transport over error prone channels may result in loss or erroneous decoding of the video. Error concealment is an effective mechanism to reconstruct the video content. In this paper, we review different error concealment methods and introduce a new framework, which we refer to as second-generation error concealment. All the error concealment methods reconstruct the lost video content by making use of some a priori knowledge about the video content. First-generation error concealment builds such a priori in a heuristic manner. The proposed second-generation error concealment builds the a priori by modeling the statistics of the video content. Context-based models are trained with the correctly decoded video content, and then used to replenish the lost video content. Trained models capture the statistics of the video content and thus reconstruct the lost video content better than reconstruction by heuristics.

A protein molecule is made of a long chain of amino acid sequences that fold into a complex three-dimensional structure. It is often the geometrical shapes that determine the protein functions. In molecular biology, researchers use sequence alignment and structure matching to compare the similarity among proteins. Considering proteins as 3D structures, we propose a novel algorithm to identify geometry-based features to retrieve similar proteins without having to deal with complex chemical characteristics and biological properties. A Web-based Dali server, which performs well in three-dimensional structure matching, is used as the ground truth to evaluate our algorithm. Our system performs close to the ground truth with much simplicity and efficiency.

Principle component analysis (PCA) has been widely used for analyzing the statistics of data. While applied to biometrics as a classification scheme, PCA faces certain challenges. We present a number of modifications to PCA in order to meet these challenges. Using face recognition as an example, we show how eigenflow, PCA applied to optimal flow, enables us to measure the difference between two images while allowing expression changes and registration error. We show how PCA can be updated to model time-varying statistics. We also show that PCA can be used to model the surface reflectance of human faces and reduce illumination variation that defeats most existing face recognition algorithms. Finally, we present distinguishing component analysis (DCA) and apply it to multimodal biometric authentication.

Proposes a method to retrieve sketches stored in the form of multiple strokes, by extracting the shape information for each stroke and by considering the spatial relation between the strokes. The spatial relation between the strokes can be used to determine the stroke correspondence between the query sketch and the database sketch. Moreover, the similarity between the relationship among query strokes and database strokes can be used as part of the matching score. Sketch retrieval is useful in various applications, such as retrieval of free-form hand-drawings, hand-written or printed text, and trademarks. For freeform hand-drawings, such as lecture notes captured digitally from a whiteboard, a user can easily search through a database of hand-drawings by inputting a sketch about what he/she is looking for, without the trouble of describing it using keywords. For hand-written or printed text, such as a protest sign in a video scene, sketch retrieval can be used to recognize the text without the need to first identify each character. For trademarks, such as company logos, sketch retrieval is useful in adding automation to the feature extraction and the retrieval process.

We present a new statistical modeling technique called "updating mixture of principal components" (UMPC). UMPC specifically captures the non-stationary as well as the multi-modal characteristics of the data. Real-world data such as video data typically have these two characteristics. The video content changes over time and has a multi-modal probability distribution. We apply UMPC to perform error concealment for video data transmitted over networks with losses, and show that UMPC outperforms conventional error concealment methods.

We introduce an efficient statistical modeling technique called mixture of principal components (MPC). This model is a linear extension to the traditional principal component analysis (PCA) and uses a mixture of eigenspaces to capture data variations. We use the model to capture face appearance variations due to pose and lighting changes. We show that this more efficient modeling leads to improved face recognition performance.

In this paper we propose a method to retrieve trademarks using query by sketches. The trademark images are first filtered to remove noise. Then each filtered image is segmented into regions based on pixel connectivity. For each region, a decision is made about whether thinning or edge extraction should be applied. Afterwards stroke tracing is performed to extract the sketch of the trademark. The user can then provide a query sketch that will be compared with those extracted sketches from the database trademark images in order to retrieve similar trademarks. Query by sketch is useful in the trademark retrieval application since the user can search for similar trademarks by providing a rough sketch instead of keywords. Our approach saves a lot of labor because it does not require the trademark database to be manually annotated with keywords and the experimental result shows that our scheme outperforms an existing method in terms of retrieval performance.

The ability to accurately track hand movement provides new opportunities for human computer interaction (HCI). Many of today's commercial hand tracking devices based on gloves can be cumbersome and expensive. An approach that avoids these problems is to use computer vision to capture hand motion. We present a complete real-time hand tracking and 3-D modeling system based on a single camera. In our system, we extract feature points from a video stream of a hand to control a virtual hand model with 2-D global motion and 3-D local motion. The on screen model gives the user instant feedback on the estimated position of the hand. This visual feedback allows a user to compensate for the errors in tracking. The system is used for three example applications. The first application uses hand tracking and gestures to take on the role of the mouse. The second interacts with a 3D virtual environment using the 3D hand model. The last application is a virtual DJ system that is controlled by hand motion tracking and gestures.

We introduce model-based schemes for error concealment of networked video. We build appearance models for specific objects in the scene and use these models to replenish any lost information. Due to the models being designed specifically for the object, they are able to capture the statistical variations in the object appearance more effectively, thereby leading to better error-concealment performance. We examine statistical modeling techniques from the literature and introduce a new efficient and accurate linear model for data representation, called the mixture of principal components (MPC), and use these models for error concealment. We simulate lossy network conditions and show that these model-based concealment schemes outperform traditional concealment schemes across a variety of loss probabilities and bit rates for the coded video.

We propose a general active learning framework for content-based information retrieval. We use this framework to guide hidden annotations in order to improve the retrieval performance. For each object in the database, we maintain a list of probabilities, each indicating the probability of this object having one of the attributes. During training, the learning algorithm samples objects in the database and presents them to the annotator to assign attributes. For each sampled object, each probability is set to be one or zero depending on whether or not the corresponding attribute is assigned by the annotator. For objects that have not been annotated, the learning algorithm estimates their probabilities with biased kernel regression. Knowledge gain is then defined to determine, among the objects that have not been annotated, which one the system is the most uncertain. The system then presents it as the next sample to the annotator to which it is assigned attributes. During retrieval, the list of probabilities works as a feature vector for us to calculate the semantic distance between two objects, or between the user query and an object in the database. The overall distance between two objects is determined by a weighted sum of the semantic distance and the low-level feature distance. The algorithm is tested on both synthetic databases and real databases of 3D models. In both cases, the retrieval performance of the system improves rapidly with the number of annotated samples. Furthermore, we show that active learning outperforms learning based on random sampling.

Abstract Video transport over error‐prone channels may result in loss or erroneous decoding of the video data. Error concealment is an effective mechanism to reconstruct the video data. In this paper, we review error‐concealment methods and introduce a new framework, which we refer to assecond‐generation error concealment. All the error‐concealment methods reconstruct the lost video data by making use of certaina prioriknowledge about the video content.First‐generation error concealmentbuilds sucha prioriin a heuristic manner. The proposed second‐generation error concealment builds thea prioriby modeling the statistics of the video content explicitly, typically in the region of interest (ROI). Context‐based models are trained with the correctly received video data and then used to replenish the lost video data. Trained models capture the statistics of the video content and thus reconstruct the lost video data better than reconstruction by heuristics. A new dynamic model ‘updating principal components’ (UPC) is proposed as a model for second‐generation error concealment. UPC can be applied to pixel values to conceal loss of pixel data. In addition, UPC can be applied to motion vectors, which results in ‘updating eigenflows’ (U‐Eigenflow), to conceal loss of motion vectors. With UPC applied to both pixel values and motion vectors, hybrid temporal/spatial error concealment can be achieved. The proposed second‐generation error‐concealment method provides superior performances to first‐generation error‐concealment methods. Copyright © 2002 John Wiley & Sons, Ltd.

This paper presents a technique for animating a three-dimensional face model through the application of principal component analysis (PCA). Using PCA has several advantages over traditional approaches to facial animation because it reduces the number of parameters needed to describe a face and confines the facial motion to a valid space to prevent unnatural contortions. First, real data is optically captured in real time from a human subject using infrared cameras and reflective trackers. This data is analyzed to find a mean face and a set of eigenvectors and eigenvalues that are used to perturb the mean face within the range described by the captured data. The result is a set of vectors that can be linearly combined and interpolated to represent different facial expressions and animations. We also show that it is possible to map the eigenvectors of one face onto another face or to change the eigenvectors to describe new motion.

There are many mode decisions in the video coding process that are used to optimize the performance in terms of the bit rate, the speed and the quality of the decoded video. We describe a classification based scheme for making mode decisions in the video coding process. We then illustrate the performance of the scheme using I/P frame selection as an example. The performance of our scheme is measured in terms of both the bit rate and the computation complexity, across different kinds of sequences, and the results are very encouraging.

Meshes are dominantly used to represent 3D models as they fit well with graphics rendering hardware. Features such as volume, moments, and Fourier transform coefficients need to be calculated from the mesh representation efficiently. We propose an algorithm to calculate these features without transforming the mesh into other representations such as the volumetric representation. To calculate a feature for a mesh, we show that we can first compute it for each elementary shape such as a triangle or a tetrahedron, and then add up all the values for the mesh. The algorithm is simple and efficient, with many potential applications.

10.1109/ICME.2001.1237748

We demonstrate a system for indexing and retrieval of 3D models aided by active learning. We propose a new set of region-based features for 3D models. Each model is treated as a solid volume with a uniform density. Features such as the volume-surface ratio, the moment invariants and the Fourier transform coefficients are efficiently calculated from the mesh model directly. Comparable retrieval performance is achieved with other features such as the cord histogram, the 3D shape spectrum, etc. To further improve the performance, we incorporate hidden annotation into our system. We propose to use active learning to improve the annotation efficiency. We show that with active learning, the system can perform better than random annotation, and the retrieval result improves rapidly with the number of annotated samples. Moreover, relevance feedback is included in the system and combined with active learning, which provides better user-adoptive retrieval results.

We propose a method for integrating audio and visual information to enhance speech recognition in adverse environments. We train the audio hidden Markov model and the visual hidden Markov model separately, and then use a Viterbi algorithm to decode both channels in parallel. The decoding process is asynchronous between the two channels to capture the asynchronous nature of audio and visual speech. We test the proposed method using speech corrupted by various types of noise and speech with the Lombard effect.

Visual communication over low bandwidth channels, such as the telephone line, requires high compression. Motion estimation and compensation techniques play an important role in achieving high compression by reducing the temporal redundancy inherent in video sequences. Current video coding standards employ the block-matching algorithm to perform motion estimation that produces visually annoying blocking artifacts at low bit-rate. In contrast, 2-D mesh motion compensation produces blocking free prediction by generating a smooth full motion field from the set of node motion vectors using spatial interpolation. However, the assumption of a globally smooth motion field is not valid for most natural occurring scenes, where motion discontinuities exist due to moving objects. We propose a post-processing scheme that refines the set of node motion vectors obtained by the 2-D mesh motion estimation. The approach is to selectively break the smoothness constraint of 2-D mesh for patches that contain motion boundaries by changing the interpolation patterns. Experimental results show improvements in both PSNR and perceptual quality over the 2-D mesh motion compensation and a block-based standard video-coding standard H.263.

There are several parts of the video-encoding process that can be optimized for high quality, high compression ratio, and fast encoding speed. We focus on some of these encoder optimization issues. We first study motion estimation, a computationally intensive part of encoding, and propose efficient algorithms based on exploiting spatially correlated motion information. We extend our motion estimation algorithms to find motion vectors for sub-parts of a block, as allowed for in the H.263 and MPEG-4 standards, and propose an efficient two-tier decision strategy for the one-four motion-vector decision. We introduce a new measure of block similarity that can improve the inter-intra mode decision significantly. Furthermore, for certain motion-estimation strategies, this new measure can also improve the bit rate resulting from a better motion estimation. Another issue that we address is the estimation of delta motion vectors in the H.263 and MPEG-4 standards. We have implemented these optimizations in a H.263 framework and the results are encouraging.

A video encoder has to make many mode decisions in order to achieve the goal of a low bit rate, high quality, and fast implementation. We propose a general classification based approach to making such mode decisions accurately and efficiently. We first illustrate the approach using the Intra-Inter coding mode decision. We then focus on the decision to skip or code a frame for rate control of video over networks. Using the classification used approach we show improvement in the rate-distortion sense. We then extend the work to scalable video coding in choosing between scalability modes and examine the performance of our approach over error prone networks, using simulated packet losses.

We describe a novel neural network architecture, which can recognize human faces with any view in a certain viewing angle range (from left 30 degrees to right 30 degrees out of plane rotation). View-specific eigenface analysis is used as the front-end of the system to extract features, and the neural network ensemble is used for recognition. Experimental results show that the recognition accuracy of our network ensemble is higher than conventional methods such as using a single neural network to recognize faces of a specific view.

We report previous effort in expanding image and video compression to computer graphics applications, in particular those based on image-based rendering. We then introduce our progress in developing a networked immersive environment that integrates image analysis, face animation, and streaming of 3D objects, to provide a truly immersive environment, with the goal of replacing existing video conferencing platforms.

Files of 3D models are often large and hence time-consuming to retrieve from a storage device or to download from the network. Most 3D viewing applications need to obtain the entire file of a 3D model in order to display it, even when the user is interested only in a small part, or a low-resolution version, of the model. Therefore, progressive coding that enables multiresolution transmission of 3D models is desired. However, existing research in progressive coding of 3D models either focuses on progressive coding of the geometry only, or progressive coding of the texture only. In this paper, we propose a joint geometry/texture coding scheme for 3D models. With this technique, geometry and texture are coded jointly and transmitted progressively to the users. Bits that represent perceptually more important information, in terms of both geometry and texture, are transmitted before the less important bits, allowing the user to obtain the best available visual quality at any time during the transmission.

We propose a simple two-state Markov modulated AR process to model the traffic created by a variable bit rate video coder. This model performs better than currently proposed models, especially for video coded with a variable group-of-pictures (GOP) structure.

In this paper, we begin by reporting our work in lip synchronization. We then expand on this technology to introduce our recent effort in developing a networked collaborative environment that integrates image analysis, face animation, and directional sound in order to provide a truly immersive environment, with the goal of replacing existing video conferencing platforms. A related technology, streaming of 3D objects, is also introduced.

High-speed desktop computers, capable of rendering complex 3D models, are becoming more commonplace, resulting in increasing interest in virtual reality systems. Virtual Reality Modelling Language (VRML) has become the de-facto standard for the description of virtual worlds over the Internet, yet the current version (VRML97) lacks capabilities for progressive transfer. This paper describes the implementation of a browser that allows the progressive transfer and simultaneous viewing of 3D models, stored in a progressive format on a Web server, using a codec based on MPEG-4 verification model source code. The authors then implement their own codec that is progressive per-vertex rather than per-level. This approach also has the benefit that it does not alter the vertex coordinates. Finally a watermarking scheme, based on the second codec, is proposed.

In this demo, we present a technique for synthesizing the mouth movement from acoustic speech information. The algorithm maps the audio parameter set to the visual parameter set using the Gaussian mixture model and the hidden Markov model. With this technique, we can create smooth and realistic lip movements.

In this demonstration, we present a prototype networked collaborative environment. This prototype supports audiovisual communication among several terminals. The voice is transmitted over the network and used to animate 3D avatars that represent the users in a virtual environment. Intelligent interfaces create a virtual environment in which users can interact with the sense of immersion.

We review recent research that examines audio-visual integration in multimodal communication. The topics include bimodality in human speech, human and automated lip reading, facial animation, lip synchronization, joint audio-video coding, and bimodal speaker verification. We also study the enabling technologies for these research topics, including automatic facial-feature tracking and audio-to-visual mapping. Recent progress in audio-visual research shows that joint processing of audio and video provides advantages that are not available when the audio and video are processed independently.

To many people, the word "multimedia" simply means the combination of various forms of information: text, speech, music, images, graphics and video. What is often overlooked is the interaction among these forms. In this paper, we present our results in exploiting the audio-visual interaction that is very significant in multimedia communication. The applications include lip synchronization, joint audio-video coding, and person verification. We present the enabling technologies, including audio-to-visual mapping and facial image analysis, for these applications. Our results show that the joint processing of audio and video provides advantages that are not available when audio and video are studied separately.

Two dimensional mesh motion compensation produces blocking free prediction in contrast to block matching motion compensation by generating a smooth full motion field from the set of node motion vectors. This is clearly not appropriate across moving object boundaries where motion discontinuities exist. In this paper, we address this problem by proposing an adaptive interpolation scheme for patches that contain multiple motions (i.e., patches around motion boundaries). Simulation results show improvements over standard 2-D mesh motion compensation at a fraction of computational increase.

A vector valued /spl Sigma//spl Delta/ modulator is proposed for multi-input signal applications, such as color imaging. Traditional /spl Sigma//spl Delta/ modulators can be modeled with a vector quantizer in 1-D signal space. The proposed /spl Sigma//spl Delta/ modulator extends this concept to M-D vector space. An optimal noise shaping filter for the oversampled vector-valued /spl Sigma//spl Delta/ modulator is considered. Simulation results verified the improvement provided by the proposed modulator.

2-D mesh motion compensation produces blocking-free prediction in contrast to block matching motion compensation by generating a smooth full motion field from the set of node motion vectors. This is clearly not appropriate across moving object boundaries where motion discontinuities exist. In this paper, we address this problem by proposing an adaptive interpolation scheme for patches that contain multiple motions (i.e. patches around motion boundaries). Simulation results show improvements over standard 2-D mesh motion compensation at a fraction of computational increase.

Analysis and classification of the scene content of a video sequence are very important for content-based indexing and retrieval of multimedia databases. We report our research on using the associated audio information for video scene classification. We describe several audio features that have been found effective in distinguishing audio characteristics of different scene classes. Based on these features, a neural net classifier was quite successful in separating audio clips from different TV programs.

One emerging application which exploits the correlation between audio and video is speech driven facial animation. The goal of speech driven facial animation is to synthesize realistic video sequences from acoustic speech. Much of the previous research has implemented this audio to visual conversion strategy with existing techniques such as vector quantization and neural networks. We examine how this conversion process can be accomplished with hidden Markov models.

We propose a scheme for coding subregions in video scenes to provide content-based scalable video. For each region, a special color is used to represent the nonobject area, and the resulting frames are coded using conventional video coding algorithms. At the decoder, the region shape is recovered based on chroma keying, and hence, content-based manipulations are made possible. A number of techniques that eliminate boundary artifacts common to region-based coding are presented. In this scheme, no explicit shape coding is needed, and advantages of existing coding algorithms are retained. This scheme was submitted to ISO MPEG-4 and performed very well in the subjective tests.

We present a novel means for predicting the shape of a person's mouth from the corresponding speech signal and explore applications of this prediction to video coding. The prediction is accomplished by modeling the probability distribution of the audiovisual features by a Gaussian mixture density. The optimal estimate for the visual features given the acoustic features can then be computed using this probability distribution. The ability to predict a person's mouth shape from the corresponding audio leads to a number of interesting joint audio-video coding strategies. In the cross-modal predictive coding system described, a model-based video coder compares measured visual parameters with predicted visual parameters, and sends the difference between the two to the receiver. Since the decoder also receives the acoustic data, it can form the prediction and then reconstruct the original parameters by adding the transmitted error signal.

We propose to use video-codec framework for the compression of volumetric medical images by treating interslice variations as "motion" of the tissues. As compression is necessary for telemedicine and archive, such an approach needs only slight modifications of existing codecs and is able to re-use most of the existing software and hardware. This is our main motivation and we seek modifications of the MPEG codec. An obvious modification is to replace block matching (BM) by more sophisticated methods. Specifically, we use the affine transformation (AT) on triangular patches. We implemented and compared our scheme with a simplified MPEG-1 encoder under identical parameters on a set of MRI images of a human hand. The blockiness is reduced dramatically and structures are better preserved. The images also appear less granular and show improvements of several dB in SNR. We expect further improvements when more features of MPEG codec are incorporated in our ongoing work.

We propose a scheme for coding sub-regions in video scenes to provide content-based scalable video. For each region, a special color is used to represent the non-object area, and the resulting frames are coded using conventional video coding algorithms. At the decoder, the region shape is recovered based on chroma keying, and hence content-based manipulations are made possible. A number of techniques that eliminate boundary artifacts common to region-based coding are presented. In this scheme, no explicit shape coding is needed, and the advantages of existing coding algorithms are retained. This scheme was submitted to ISO MPEG-4, and performed very well in the subjective tests.

Foreground and background features are focused (or defocused) differently in an image because corresponding objects are at different depths in the scene. This paper presents a novel approach for segmenting foreground and background in video images based on feature defocus. A modified defocus measurement that distinguishes between high-contrast defocused edges and low-contrast focused edges is presented. Defocus-based segmentation is desirable because defocus techniques are computationally simple. Results indicate that the foreground is easily segmented from moving background. This approach, coupled with motion detection, can segment complex scenes containing both moving background and stationary foreground.

We utilize speech information to improve the quality of audio-visual communications such as video telephony and videoconferencing. We show that the marriage of speech analysis and image processing can solve problems related to lip synchronization. We present a technique called speech-assisted frame-rate conversion, and apply it to coding of talking head video. Demonstration sequences are presented. Extensions and other applications are outlined.

A video codec typically-skips frames to satisfy the bit rate constraint. This results in a jerky motion and a loss of lip synchronization in sequences of talking persons. We propose a frame interpolation technique to solve these problems. The main components of this technique include: foreground/background segmentation, mesh-based frame interpolation, and image analysis/synthesis for mouth movements. This technique generates smooth head motion and renders lip motion that is synchronized with the voice of the person.

Anticausal inversion of IIR transfer functions has gained importance in recent years, in the efficient implementation of IIR digital filter banks. In this paper we first introduce the idea of a causal dual, as an intermediate step in the implementation of anticausal IIR inverses. With time reversal operators at the input and output of the causal dual, we get the anticausal inverse of the original structure. The causal dual eliminates the need for similarity transformations, during a key step called blockwise state transfer, in implementing anticausal inverses. In the paper we identify efficient structures for causal duals of standard structures like the direct-form, cascade-form, coupled form, and IIR lattice structures, including the tapped lattice.

We utilize speech information to improve the quality of audio-visual communications such as videotelephony and videoconferencing. In particular, the marriage of speech analysis and image processing can solve problems related to lip synchronization. We present a technique called speech-assisted frame-rate conversion. Demonstration sequences are presented. Other applications, including speech-assisted video coding, are outlined.>

Analysis/synthesis filter banks with perfect reconstruction property have attracted much attention. In some applications, particularly image coding, it is desirable to design filter banks in which the filters have linear phase. We present a method of linearizing the passband phase response of filters in any given filter bank. While linearizing the phase response, this method preserves the magnitude response of the filters, the perfect reconstruction property, and the paraunitary property (orthonormality). We present the eigen-approach design and the anticausal implementation of this technique. Using this technique, we illustrate the importance of linear phase in subband coding.>

Subband image coding produces several types of artifacts, including contouring and ringing. We use the dithering technique to remove these artifacts. By dithering the subband signals, we can replace signal-dependent artifacts with signal-independent white noise which is less annoying to human eyes. The subjective quality of the coded images is hence improved. In addition, signal-independent white noise can be easily removed by existing noise-reduction methods. The dithering technique can also be applied to transform coding, including the discrete cosine transform, to remove the blocking artifact. Experiment results that illustrate the effectiveness of dithering are given.>

In this paper, we present a new blocking scheme for multidimensional transform coding. This scheme generalizes traditional blocking methods. Compared to the conventional rectangular blocking method, the generalized blocking scheme provides us more freedom in choosing block shapes and arrangements, so larger coding gain can be achieved. Examples are presented to illustrate the improvement given by the proposed scheme.>

The authors review the fundamentals of multidimensional (M-D) multirate signal processing and present recent and new developments in M-D multirate systems. A method is presented for deriving all parallelepiped-shaped filters in M-D multirate applications from an appropriate 1-D filter. It is shown that the generalized-pseudocirculant property is necessary and sufficient for an M-D maximally decimated filter-bank system to be free from aliasing. It is shown how the Smith form, Smith-McMillan form, and the least common multiples of integer matrices can resolve many nonseparable M-D multirate problems. The condition for alias-free decimation and the multistage design of decimation systems are mentioned.>

Multidimensional (MD) multirate systems, which find applications in the coding and compression of image and video data, have attracted much attention. The basic building blocks in a MD multirate system are the decimation matrix M, the expansion matrix L, and MD digital filters. With D denoting the number of dimensions, M and L are D*D nonsingular integer matrices. When these matrices are diagonal, most of the one-dimensional multirate results can be extended automatically. However, for the nondiagonal case, these extensions are nontrivial. One example of this nature is the commutativity of MD decimation and expansion matrices. Using the concepts of coprimeness and least common right/left multiples of integer matrices, a set of necessary and sufficient conditions is derived for a decimation matrix and an expansion matrix to commute. This commutativity is also used to derive an efficient polyphase implementation of an MD decimation system with rational decimation matrix.>