Lack of human prior knowledge is one of the main reasons that semantic gap still remains when it comes to automatic multimedia understanding. In this work, we exploit the ontological structure of target concepts and propose an universal ontological inference framework for image understanding. The framework explicitly utilizes subclass and co-occurrence relation to effectively refine the coarse concept detections. Moreover, we show how to automatically construct and learn the underlying ontology required by the framework. As can be shown by experiments, the result is an effective and robust algorithm that characterizes well the structure of the target concepts and outperforms the state-of-the-art methods.

The abundance of multimedia data on the Web presents both challenges (how to annotate, search, and mine) and opportunities (crawling the Web to create large structured multimedia data bases which can be used to do inference effectively). Because of the huge data volume, considering all semantic concepts as on the same (flat) level is not viable. In this paper, we introduce a unified STRUCTURED representation called multimedia information networks (MINets), which incorporates ontology and cross-media links, covering both content and context knowledge. Ontology and cross-media structures are constructed and expanded by automatically constructing MINets from web-scale data by state-of-the-art information extraction and knowledge-based population techniques. The resultant MINet will contain a wide range of linkages, including logical, statistical, and semantic relations among informative concept nodes, which connects proliferative ontology as well as cross-media web-scale resources together. The raw data collected in construction phase often contain much noisy, incomplete, or even conflicting information which could be detrimental to information extraction and utilization. Then, the redundant link structure can be utilized to distill MINets and improve quality of information (QoI). Moreover, advanced inference theory and system can be built upon the linked MINets, and then high-level ontological knowledge can be inferred and integrated in a logically harmonious network structure in MINets which is consistent with human cognition. Even more, as information channels, the ontology and cross-media links in MINets connect informative knowledge resources together, which makes it possible to increase the portability of information between different resources to increase information utilization levels.

We explore event recognition in the personal photo album setting where the task is not to identify event in individual photo but in the whole album. This setting arises from the way people organize their own collections and the fact that individual photo in these albums often fails to convey meaningful event semantic behind the the album. We work on this problem in a object-centric manner, i.e. we train detectors for objects relevant to the events in the holiday dataset we built, and then detect these holidays based on object detector outputs. The prior knowledge, i.e. what objects are relevant to the event, is obtained statistically from mass image collection web site and thus tends to be more accurate and less biased.

In this paper, we study the problem of recognizing events in personal photo albums. In consumer photo collections or online photo communities, photos are usually organized in albums according to their events. However, interpreting photo albums is more complicated than the traditional problem of understanding single photos, because albums generally exhibit much more varieties than single image. To solve this challenge, we propose a novel representation, called Compositional Object Pattern, which characterizes object level pattern conveying much richer semantic than low level visual feature. To interpret the rich semantics in albums, we mine frequent object patterns in the training set, and then rank them by their discriminating power. The album feature is then set as the frequencies of these frequent and discriminative patterns, called Compositional Object Pattern Frequency(COPF). We show with experimental result that our algorithm is capable of recognizing holidays with accuracy higher than the baseline method.

In the field of machine learning and pattern recognition, an alternative to conventional classification is hierarchical classification that exploits hierarchical relations between concepts of interest. To the best of our knowledge, all hierarchical classification methods in the literature are designed to reduce computation complexity without sacrificing too much on accuracy performance. In this work on image classification, we first propose a hierarchical image feature extraction that extracts image feature based on the location of current node in hierarchy to fit the images under current node and to better distinguish its subclasses. As far as we know, such node-dependent feature extraction has not been considered in the literature. Contrary to former hierarchical classification methods that only consider local structure of the hierarchy, we propose a novel cross-level hierarchical classification method that utilizes both global and local concept structures throughout the entire path decision-making process. Our experimental result on two datasets shows that the proposed hierarchical feature extraction combined with our novel hierarchical classification achieves better accuracy performance than conventional non-hierarchical classification methods, and hence conventional hierarchical methods as well.

Image is a powerful medium for expressing one's ideas and rightly confirms the adage, “One picture is worth a thousand words.” In this work, we explore the application of world knowledge in the form of Wikipedia to achieve this objective-literally. In the first part, we disambiguate and rank semantic concepts associated with ambiguous keywords by exploiting link structure of articles in Wikipedia. In the second part, we explore an image representation in terms of keywords which reflect the semantic content of an image. Our approach is inspired by the desire to augment low-level image representation with massive amounts of world knowledge, to facilitate computer vision tasks like image retrieval based on this information. We represent an image as a weighted mixture of a predetermined set of concrete concepts whose definition has been agreed upon by a wide variety of audience. To achieve this objective, we use concepts defined by Wikipedia articles, e.g., sky, building, or automobile. An important advantage of our approach is availability of vast amounts of highly organized human knowledge in Wikipedia. Wikipedia evolves rapidly steadily increasing its breadth and depth over time.

We investigate effective means of building robust dictionaries for detecting the sparse foreground in videos with static background. This work is an extension to our existing solution to foreground/background segmentation problem using the linear programming method proposed to detect sparse errors in signals, which are created by a known dictionary. The dictionary building methods we study are established robust component analysis techniques in the literature (i.e. k-SVD & robust-PCA) as well as a heuristic (running median) inspired by the highly correlated nature of the static video background signal. We compare the effectiveness of the new methods with our original system as well as a baseline method, which is the commonly used single Gaussian model of the background pixels.

We consider the problem of recognizing human faces despite variations in both pose and illumination, using only frontal training images. We propose a very simple algorithm, called nearest-subspace patch matching, which combines a local translational model for deformation due to pose with a linear subspace model for lighting variations. This algorithm gives surprisingly competitive performance for moderate variations in both pose and illumination, a domain that encompasses most face recognition applications, such as access control. The results also provide a baseline for justifying the use of more complicated face models or more advanced learning methods to handle more extreme situations. Extensive experiments on publicly available databases verify the efficacy of the proposed method and clarify its operating range.