In the intricate problem of understanding long-form multi-modal inputs, few key-aspects in scene-understanding and dialogue-and-discourse are often overlooked. In this paper, we investigate two such key-aspects for better semantic and relational understanding - (i). head-object-tracking in addition to usual face-tracking, and (ii). fusing scene-to-text representation with external common-sense knowledge-base for effective mapping to sub-tasks of interest. The usage of head-tracking especially helps with enriching sparse entity mapping to inter-entity conversation interactions. These methods are guided by natural language supervision on visual models, and perform well for interaction and sentiment understanding tasks.

The natural language processing community has had a major interest in auto-regressive [4, 13] and span-prediction based language models [7] recently, while knowledge graphs are often referenced for common-sense based reasoning and fact-checking models. In this paper, we present an equivalence representation of span-prediction based language models and knowledge-graphs to better leverage recent developments of language modelling for multi-modal problem statements. Our method performed well, especially with sentiment understanding for multi-modal inputs, and discovered potential bias in naturally occurring videos when compared with movie-data interaction-understanding. We also release a dataset of an auto-generated questionnaire with ground-truths consisting of labels spanning across 120 relationships, 99 sentiments, and 116 interactions, among other labels for finer-grained analysis of model comparisons in the community.

We consider the problem of building semantic relationship of unseen entities from free-form multi-modal sources. This intelligent agent understands semantic properties by creating (1) logical segments from sources, (2) finds interacting objects, (3) infers their interaction actions using (4) extracted textual, auditory, visual, and tonal information. The conversational dialogue discourses are automatically mapped to interacting co-located objects, and fused with their Kinetic action embeddings at each scene of occurrence. This generates a combined probability distribution representation for interacting entities spanning over every semantic relation class. Using these probabilities, we create knowledge graphs capable of answering semantic queries and infer missing properties in a given context.

To reduce the significant redundancy in deep Convolutional Neural Networks (CNNs), most existing methods prune neurons by only considering the statistics of an individual layer or two consecutive layers (e.g., prune one layer to minimize the reconstruction error of the next layer), ignoring the effect of error propagation in deep networks. In contrast, we argue that for a pruned network to retain its predictive power, it is essential to prune neurons in the entire neuron network jointly based on a unified goal: minimizing the reconstruction error of important responses in the "final response layer" (FRL), which is the second-to-last layer before classification. Specifically, we apply feature ranking techniques to measure the importance of each neuron in the FRL, formulate network pruning as a binary integer optimization problem, and derive a closed-form solution to it for pruning neurons in earlier layers. Based on our theoretical analysis, we propose the Neuron Importance Score Propagation (NISP) algorithm to propagate the importance scores of final responses to every neuron in the network. The CNN is pruned by removing neurons with least importance, and it is then fine-tuned to recover its predictive power. NISP is evaluated on several datasets with multiple CNN models and demonstrated to achieve significant acceleration and compression with negligible accuracy loss.

Modern brain mapping techniques are producing increasingly large datasets of anatomical or functional connection patterns. Recently, it became possible to record detailed live imaging videos of mammal brain while the subject is engaging routine activity. We analyze a dataset of videos recorded from ten mice to describe how to detect neurons, extract neuron signals, map correlation of neuron signals to mice activity, detect the network topology of active neurons, and analyze network topology characteristics. We propose neuron position alignment to compensate the distortion and movement of cerebral cortex in live mouse brain and the background luminance compensation to extract and model neuron activity. To find out the network topology as an undirected graph model, a cross-correlation based method is proposed and used for analysis. Afterwards, we did preliminary analysis on network topologies. The significance of this paper is on how to extract neuron activities from live mouse brain imaging videos and a network analysis method to analyze topology that can potentially provide insight on how neurons are actively connected under stimulus, rather than analyzing static neural networks.

Users with anomalous behaviors in online communication systems (e.g. email and social medial platforms) are potential threats to society. Automated anomaly detection based on advanced machine learning techniques has been developed to combat this issue; challenges remain, though, due to the difficulty of obtaining proper ground truth for model training and evaluation. Therefore, substantial human judgment on the automated analysis results is often required to better adjust the performance of anomaly detection. Unfortunately, techniques that allow users to understand the analysis results more efficiently, to make a confident judgment about anomalies, and to explore data in their context, are still lacking. In this paper, we propose a novel visual analysis system, TargetVue, which detects anomalous users via an unsupervised learning model and visualizes the behaviors of suspicious users in behavior-rich context through novel visualization designs and multiple coordinated contextual views. Particularly, TargetVue incorporates three new ego-centric glyphs to visually summarize a user's behaviors which effectively present the user's communication activities, features, and social interactions. An efficient layout method is proposed to place these glyphs on a triangle grid, which captures similarities among users and facilitates comparisons of behaviors of different users. We demonstrate the power of TargetVue through its application in a social bot detection challenge using Twitter data, a case study based on email records, and an interview with expert users. Our evaluation shows that TargetVue is beneficial to the detection of users with anomalous communication behaviors.

In this paper, we develop a pipelining schema for image deep learning on GPU cluster to leverage heavy workload of training procedure. In addition, it is usually necessary to train multiple models to obtain a good deep learning model due to the limited a priori knowledge on deep neural network structure. Therefore, adopting parallel and distributed computing appears is an obvious path forward, but the mileage varies depending on how amenable a deep network can be parallelized and the availability of rapid prototyping capabilities with low cost of entry. In this work, we propose a framework to organize the training procedures of multiple deep learning models into a pipeline on a GPU cluster, where each stage is handled by a particular GPU with a partition of the training dataset. Instead of frequently migrating data among the disks, CPUs, and GPUs, our framework only moves partially trained models to reduce bandwidth consumption and to leverage the full computation capability of the cluster. In this paper, we deploy the proposed framework on popular image recognition tasks using deep learning, and the experiments show that the proposed method reduces overall training time up to dozens of hours compared to the baseline method.

The advances of mobile computing and sensor technology have turned the mobile devices into powerful instruments. The integration of thermal and visual cameras extends the capability of computer vision, due to the fact that both images reveal different characteristics in images, however, image alignment is a challenge. This paper proposes an effective approach to align image pairs for event detection on mobile through image recognition. We leverage thermal and visual cameras as multi-modality sources for image recognition. By analyzing the heat pattern, the proposed APP can identify the heating sources and help users inspect their house heating system, on the other hand, with applying image recognition, the proposed APP furthermore can help field workers identify the asset condition and provide the guidance to solve their issues.

With the emergence of data science, graph computing is becoming a crucial tool for processing big connected data. Although efficient implementations of specific graph applications exist, the behavior of full-spectrum graph computing remains unknown. To understand graph computing, we must consider multiple graph computation types, graph frameworks, data representations, and various data sources in a holistic way.

Many Big Data analytics essentially explore the relationship among interconnected entities, which are naturally represented as graphs. However, due to the irregular data access patterns in the graph computations, it remains a fundamental challenge to deliver highly efficient solutions for large scale graph analytics. Such inefficiency restricts the utilization of many graph algorithms in Big Data scenarios. To address the performance issues in large scale graph analytics, we develop a graph processing system called System G, which explores efficient graph data organization for parallel computing architectures. We discuss various graph data organizations and their impact on data locality during graph traversals, which results in various cache performance behavior on processor side. In addition, we analyze data parallelism from architecture's perspective and experimentally show the efficiency for System G based graph analytics. We present experimental results for commodity multicore clusters and IBM PERCS supercomputers to illustrate the performance of System G for large scale graph analytics.

Understanding how human emotion is evoked from visual content is a task that we as people do every day, but machines have not yet mastered. In this work we address the problem of predicting the intended evoked emotion at given points within movie trailers. Movie Trailers are carefully curated to elicit distinct and specific emotional responses from viewers, and are therefore well-suited for emotion prediction. However, current emotion recognition systems struggle to bridge the "affective gap", which refers to the difficulty in modeling high-level human emotions with low-level audio and visual features. To address this problem, we propose a mid-level concept feature, which is based on detectable movie shot concepts which we believe to be tied closely to emotions. Examples of these concepts are "Fight", "Rock Music", and "Kiss". We also create 2 datasets, the first with shot-level concept annotations for learning our concept detectors, and a separate, second dataset with emotion annotations taken throughout the trailers using the two dimensional arousal and valence model for emotion annotation. We report the performance of our concept detectors, and show that by using the output of these detectors as a mid-level representation for the movie shots we are able to more accurately predict the evoked emotion throughout a trailer than by using low-level features.

We propose a personality detection algorithm based on human perceptions of images and created a mobile social application named Date gram to test the efficiency of our algorithm and connect people who share similar image perceptions. We detect user's Big Five personality based on their perceptual opinions on several images. Users in Date gram are matched to play visual-sentiment games, through which their connections based on personality similarity are further strengthened. Datagram utilizes techniques of image sentiment detection to analyze user's personality. Given the collected personal sentiment perception data, we proposed personality-mapping algorithm to categorize users into "Big Five" personality. After the Adjective-Noun-Pairs (ANPs) are extracted from each image, a personalized image-profile that consists of multiple pairs of ANPs and corresponding user decisions is generated. This profile is treated as the input of our algorithm, and a modified Latent Dirichlet Allocation (LDA) model is applied to extract personality-related features. "Big Five" personality values are calculated by a word-personality mapping algorithm where the technique of Linguistic Inquiry and Word Count (LIWC) is used. To evaluate our proposed algorithm, we did user studies over visual sentiments as well as personality detections. The results demonstrate that sentimental opinions from different persons vary significantly towards the same image, people would prefer to connect with those sharing similar perceptions of images and similar personalities with them.

Successful enterprises depend on high performing teams consisting of productive individuals, who can effective find valuable information. Predictive methods are highly desired to identify these inter-related, multi-typed entities that can potentially help to improve enterprise performance before they become productive. For such predictive methods, observation of their dynamic behavior in the organizational social networks may provide useful input. In this paper, we propose a novel approach to analyze and rank heterogeneous objects in multi-level networks by their value for improving productivity. Compared to existing approaches that either focus on static factors of productivity or single type of entities (e.g., individuals), our work offers two unique contributions. First, we propose a novel multi-level synchronicity network representation which allows us to exploit the structural characteristics of various entities’ dynamic behavior. Furthermore, based on the synchronicity networks, we propose a novel algorithm for Heterogeneous Multi-level networks Ranking, to simultaneously rank inter-related heterogeneous entities (e.g., topics, individuals and teams) by their value. Our experiments demonstrate that our approach significantly outperforms existing methods in both enterprise organizational social networks and public social media.

A recent topic of research in the network science community is combined or composite networks - these are two or more interacting networks that must be characterized jointly rather than individually. For example, a social network and a communication network sharing some nodes (corresponding to users) may be modeled together as a composite network. This may be useful since the aggregate performance of a composite network may often depend on how the individual networks influence each other. Information may travel faster or slower through composite networks depending on how they are coupled. The above vision was captured in the Call for Papers for this special issue in the IEEE Journal On Selected Areas In Communications, and it was published in June 2012. As a result of this solicitation, we received 62 submissions by the deadline of August 15, 2012. Papers were selected after two rigorous rounds of review. The first round of notifications were sent out on December 17, 2012 to the authors whose papers passed the first round of review. Revised versions of the papers were submitted on January 31, 2013. Finally, after a second round of review, the Guest Editorial board decided on March 12, 2013 to accept 16 high quality papers for publication in this competitive special issue. Papers appearing in this special issue belong to five broad themes, which are not necessarily mutually exclusive: (1) fundamental principles in network science, (2) information propagation models in networks, (3) bringing insights from other genres of networks, (4) economic and game theoretic models, and (5) application of network science principles to communications networking problems.

Term suggestions recommend query terms to a user based on his initial query. Suggesting adequate terms is a challenging issue. Most existing commercial search engines suggest search terms based on the frequency of prior used terms that match the leading alphabets the user types. In this article, we present a novel mechanism to construct semantic term-relation graphs to suggest relevant search terms in the semantic level. We built term-relation graphs based on multipartite networks of existing social media, especially from Wikipedia. The multipartite linkage networks of contributor-term, term-category, and term-term are extracted from Wikipedia to eventually form term relation graphs. For fusing these multipartite linkage networks, we propose to incorporate the contributor-category networks to model the expertise of the contributors. Based on our experiments, this step has demonstrated clear enhancement on the accuracy of the inferred relatedness of the term-semantic graphs. Experiments on keyword-expanded search based on 200 TREC-5 ad-hoc topics showed obvious advantage of our algorithms over existing approaches.

Sentiment is a rich and important dimension of social interaction. However, its presence in computer-mediated communication in corporate settings is not well understood. This paper provides a preliminary study of people's expression of sentiment in email conversations in an organizational context. The study reveals that sentiment levels evolve over time during the process of newcomers' socialization, that sentiment varies according to tie-strength with the recipient, and that sentiment patterns can be indicative of one's position in the corporate social network as well as job performance. These findings shed light on the complex and dynamic nature of sentiment patterns, and would inspire further explorations and applications of sentiment analysis in organizations.

Social network analysis (SNA) has been a research focus in multiple disciplines for decades, including sociology, healthcare, business management, etc. Traditional SNA researches concern more human and social science aspects—trying to undermine the real relationship of people and the impacts of these relationships. While online social networks have become popular in recent years, social media analysis, especially from the viewpoint of computer scientists, is usually limited to the aspects of people's behavior on specific websites and thus are considered not necessarily related to the day-to-day people's behavior and relationships. We conduct research to bridge the gap between social scientists and computer scientists by exploring the multifacet existing social networks in organizations that provide better insights on how people interact with each other in their professional life. We describe a comprehensive study on the challenges and solutions of mining and analyzing existing social networks in enterprise. Several aspects are considered, including system issues; privacy laws; the economic value of social networks; people's behavior modeling including channel, culture, and social inference; social network visualization in large-scale organization; and graph query and mining. The study is based on an SNA tool (SmallBlue) that was designed to overcome practical challenges and is based on the data collected in a global organization of more than 400 000 employees in more than 100 countries.

Nonnegative Matrix Factorization (NMF) techniques has aroused considerable interests from the field of artificial intelligence in recent years because of its good interpretability and computational efficiency. However, in many real world applications, the data features usually evolve over time smoothly. In this case, it would be very expensive in both computation and storage to rerun the whole NMF procedure after each time when the data feature changing. In this paper, we propose Evolutionary Nonnegative Matrix Factorization (eNMF), which aims to incrementally update the factorized matrices in a computation and space efficient manner with the variation of the data matrix. We devise such evolutionary procedure for both asymmetric and symmetric NMF. Finally we conduct experiments on several real world data sets to demonstrate the efficacy and efficiency of eNMF.

In recommendation systems, a central host typically requires access to user profiles in order to generate useful recommendations. This access, however, undermines user privacy; the more information is revealed to the host, the more the user's privacy is compromised. In this paper, we propose a novel end-to-end encrypted recommendation mechanism which encrypts sensitive private data at the user end, without ever exposing plaintext private data to the host server. Unlike previously proposed privacy-preserving recommendation mechanisms, the data in this proposed system are lossless - a pivotal feature to many applications, e.g., in health informatics, business analytics, cyber security, etc. We achieve this goal by developing encrypted-domain polynomial ring homomorphism cryptographic algorithms to compute similarity of encrypted scores on the server, so that collaborative recommendations can be computed in the encryption domain and only an authorized person can decrypt the exact results. We also propose a novel key management system to make sure private information retrieval and recommendation computations can be executed in the encrypted domain in practice. Our experiments show that the proposed scheme offers robust security and lossless accurate recommendation, as well as high efficiency. Our preliminary results show the recommendation accuracy is 21% better than the existing statistical lossy privacy-preserving mechanisms based on random perturbation and user profile distribution. This new approach can potentially be applied to various data mining and cloud computing environments and significantly alleviates the privacy concerns of users.

Prior research has provided some evidence of social correlation (i.e., "you are who you know"), which makes it possible to infer one's interests from his or her social neighbors. However, it is also shown to be challenging to consistently obtain high quality inference. This challenge can be partially attributed to the fact that people usually maintain diverse social relationships, in order to tap into diverse information and knowledge. It is unlikely that a person would possess all interests of his/her social neighbors. Instead, s/he may selectively acquire just a subset of them. This paper intends to improve inferring interests from neighbors given this observation. We conduct this study by implementing a privacy-preserving large distributed social sensor system in a large global IT company to capture the multifaceted activities (e.g., emails, instant messaging, social bookmarking, etc.) of 25K+ people. These activities occupy the majority of employees' time, and thus, provide a higher quality view of the diverse aspects of their professional interests compared to the friending activity on online social networking sites. In this paper, we propose a technique that exploits the correlation among the attributes that a person possesses to improve social-correlation-based inference quality. Our technique offers two unique contributions. First, we demonstrate that the proposed technique can significantly improve inference quality by as much as 76.1%. Second, we study the interaction between the two factors: social correlation and attribute correlation under different situations. The results can inform practical applications how the inference quality would change in various scenarios.

In this paper we feature a set of research projects done at several IBM Research laboratories across the world. The work featured here focuses on the topic of smart social collaboration, which studies, designs, and develops social collaboration principles and technologies that can help customize and enhance existing social collaboration tools to suit specific user needs, including cultural, business, and personal needs.

Real-world social networks are dynamic in nature. Companies continue to collaborate, align strategically, acquire, and merge over time, and receive positive/negative impact from other companies. Consequently, their performance changes with time. If one can understand what types of network changes affect a company's value, he/she can predict the future value of the company, grasp industry innovations, and make business more successful. However, it often requires continuous records of relational changes, which are often difficult to track for companies, and the models of mining longitudinal network are quite complicated. In this study, we developed algorithms and a system to infer large-scale evolutionary company networks from public news during 1981-2009. Then, based on how networks change over time, as well as the financial information of the companies, we predicted company profit growth. This is the first study of longitudinal network-mining-based company performance analysis in the literature

Graphs appear in numerous applications including cyber-security, the Internet, social networks, protein networks, recommendation systems, and many more. Graphs with millions or even billions of nodes and edges are common-place. How to store such large graphs efficiently? What are the core operations/queries on those graph? How to answer the graph queries quickly? We propose GBASE, a scalable and general graph management and mining system. The key novelties lie in 1) our storage and compression scheme for a parallel setting and 2) the carefully chosen graph operations and their efficient implementation. We designed and implemented an instance of GBASE using MapReduce/Hadoop. GBASE provides a parallel indexing mechanism for graph mining operations that both saves storage space, as well as accelerates queries. We ran numerous experiments on real graphs, spanning billions of nodes and edges, and we show that our proposed GBASE is indeed fast, scalable and nimble, with significant savings in space and time.

Diversified ranking on graphs is a fundamental mining task and has a variety of high-impact applications. There are two important open questions here. The first challenge is the measure - how to quantify the goodness of a given top-k ranking list that captures both the relevance and the diversity? The second challenge lies in the algorithmic aspect - how to find an optimal, or near-optimal, top-k ranking list that maximizes the measure we defined in a scalable way? In this paper, we address these challenges from an optimization point of view. Firstly, we propose a goodness measure for a given top-k ranking list. The proposed goodness measure intuitively captures both (a) the relevance between each individual node in the ranking list and the query; and (b) the diversity among different nodes in the ranking list. Moreover, we propose a scalable algorithm (linear wrt the size of the graph) that generates a provably near-optimal solution. The experimental evaluations on real graphs demonstrate its effectiveness and efficiency.

Given an IP source-destination traffic network, how do we spot mis-behavioral IP sources (e.g., port-scanner)? How do we find strange users in a user-movie rating graph? Moreover, how can we present the results intuitively so that it is relatively easier for data analysts to interpret? We propose NrMF, a non-negative residual matrix factorization framework, to address such challenges. We present an optimization formulation as well as an effective algorithm to solve it. Our method can naturally capture abnormal behaviors on graphs. In addition, the proposed algorithm is linear wrt the size of the graph therefore it is suitable for large graphs. The experimental results on several data sets validate its effectiveness as well as efficiency.

Information spreading processes are central to human interactions. Despite recent studies in online domains, little is known about factors that could affect the dissemination of a single piece of information. In this paper, we address this challenge by combining two related but distinct datasets, collected from a large scale privacy-preserving distributed social sensor system. We find that the social and organizational context significantly impacts to whom and how fast people forward information. Yet the structures within spreading processes can be well captured by a simple stochastic branching model, indicating surprising independence of context. Our results build the foundation of future predictive models of information flow and provide significant insights towards design of communication platforms.

Image annotation based on visual features has been a difficult problem due to the diverse associations that exist between visual features and human concepts. In this paper, we propose a novel approach called Annotation by Image-to-Concept Distribution Model (AICDM) for image annotation by discovering the associations between visual features and human concepts from image-to-concept distribution. Through the proposed image-to-concept distribution model, visual features and concepts can be bridged to achieve high-quality image annotation. In this paper, we propose to use “visual features”, “models”, and “visual genes” which represent analogous functions to the biological chromosome, DNA, and gene. Based on the proposed models using entropy, tf-idf, rules, and SVM, the goal of high-quality image annotation can be achieved effectively. Our empirical evaluation results reveal that the AICDM method can effectively alleviate the problem of visual-to-concept diversity and achieve better annotation results than many existing state-of-the-art approaches in terms of precision and recall.

Up to the present, the contemporary studies are not really successful in image annotation due to some critical problems like diverse regularities between visual features and human concepts. Such diverse regularities make it hard to annotate the image semantics correctly. In this paper, we propose a novel approach called AICDM (Annotation by Image-Concept Distribution Model) for image annotation by discovering the associations between visual features and human concepts from image-concept distribution. Through the proposed image-concept distribution model, the uncertain regularities between visual features and human concepts can be clarified for achieving high-quality image annotation. The empirical evaluation results also reveal that our proposed AICDM method can effectively alleviate the uncertain regularity problem and bring out better annotation results than other existing approaches in terms of precision and recall.

This paper intends to provide some insights of a scientific problem: how likely one's interests can be inferred from his/her social connections -- friends, friends' friends, 3-degree friends, etc? Is "Birds of a Feather Flocks Together" a norm? We do not consider the friending activity on online social networking sites. Instead, we conduct this study by implementing a privacy-preserving large distribute social sensor system in a large global IT company to capture the multifaceted activities of 30,000+ people, including communications (e.g., emails, instant messaging, etc) and Web 2.0 activities (e.g., social bookmarking, file sharing, blogging, etc). These activities occupy the majority of employees' time in work, and thus, provide a high quality approximation to the real social connections of employees in the workplace context. In addition to such "informal networks", we investigated the "formal networks", such as their hierarchical structure, as well as the demographic profile data such as geography, job role, self-specified interests, etc. Because user ID matching across multiple sources on the Internet is very difficult, and most user activity logs have to be anonymized before they are processed, no prior studies could collect comparable multifaceted activity data of individuals. That makes this study unique. In this paper, we present a technique to predict the inference quality by utilizing (1) network analysis and network autocorrelation modeling of informal and formal networks, and (2) regression models to predict user interest inference quality from network characteristics. We verify our findings with experiments on both implicit user interests indicated by the content of communications or Web 2.0 activities, and explicit user interests specified in user profiles. We demonstrate that the inference quality prediction increases the inference quality of implicit interests by 42.8%, and inference quality of explicit interests by up to 101%.

Enterprises depend on their information workers finding valuable information to be productive. However, existing enterprise search and recommendation systems can exploit few studies on the correlation between information content and information workers' productivity. In this paper, we combine content, social network and revenue analysis to identify computational metrics for finding valuable information content in people's electronic communications within a large-scale enterprise. Specifically, we focus on two questions: (1) how are the topics extracted from such content correlate with information workers' performance? and (2) how to find valuable topics with potentially high impact on employee performance? For the first question, we associate the topics with the corresponding workers' productivity measured by the revenue they generate. This allows us to evaluate the topics' influence on productivity. We further verify that the derived topic values are consistent with human assessor subjective evaluation. For the second question, we identify and evaluate a set of significant factors including both content and social network factors. In particular, the social network factors are better in filtering out low-value topics, while content factors are more effective in selecting a few top high-value topics. In addition, we demonstrate that a Support Vector regression model that combines the factors can already effectively find valuable topics. We believe that our results provide significant insights towards scientific advances to find valuable information.

Search and recommendation systems must effectively model user interests in order to provide personalized results. The proliferation of social software makes social network an increasingly important source for user interest modeling, because of the social influence and correlation among friends. However, there are large variations in people's contribution of social content. Therefore, it is impractical to accurately model interests for all users. As a result, applications need to decide whether to utilize a user interest model based on its accuracy. To address this challenge, we present a study on the accuracy of user interests inferred from three types of social content: social bookmarking, file sharing, and electronic communication, in an organizational social network within a large-scale enterprise. First, we demonstrate that combining different types of social content to infer user interests outperforms methods that use only one type of social content. Second, we present a technique to predict the inference accuracy based on easily observed network characteristics, including user activeness, network in-degree, out-degree, and betweenness centrality.

Visualizing large-scale online social network is a challenging yet essential task. This paper presents HiMap, a system that visualizes it by clustered graph via hierarchical grouping and summarization. HiMap employs a novel adaptive data loading technique to accurately control the visual density of each graph view, and along with the optimized layout algorithm and the two kinds of edge bundling methods, to effectively avoid the visual clutter commonly found in previous social network visualization tools. HiMap also provides an integrated suite of interactions to allow the users to easily navigate the social map with smooth and coherent view transitions to keep their momentum. Finally, we confirm the effectiveness of HiMap algorithms through graph-travesal based evaluations.

SmallBlue is a social networking application that unlocks the valuable business intelligence of 'who knows what?', 'who knows whom?' and 'who knows what about whom' within an organization, without requiring explicit involvement of individuals. The aim of SmallBlue is to locate knowledgeable colleagues, communities, and knowledge networks in companies. The suite also helps users manage their personal networks, and reach out to their extended network (the friends of their friends) to find and access expertise and information.

Audio is a useful modality complement to video for healthcare monitoring. In this paper, we investigate the use of Hierarchical Hidden Markov Models (HHMMs) for healthcare audio event classification. We show that HHMM can handle audio events with recursive patterns to improve the classification performance. We also propose a model fusion method to cover large variations often existing in healthcare audio events. Experimental results from classifying key eldercare audio events show the effectiveness of the model fusion method for healthcare audio event classification.

With the explosion of social media, scalability becomes a key challenge. There are two main aspects of the problems that arise: 1) data volume: how to manage and analyze huge datasets to efficiently extract patterns, 2) data understanding: how to facilitate understanding of the patterns by users? To address both aspects of the scalability challenge, we present a hybrid approach that leverages two complementary disciplines, data mining and information visualization. In particular, we propose 1) an analytic data model for content-based networks using tensors; 2) an efficient high-order clustering framework for analyzing the data; 3) a scalable context-sensitive graph visualization to present the clusters. We evaluate the proposed methods using both synthetic and real datasets. In terms of computational efficiency, the proposed methods are an order of magnitude faster compared to the baseline. In terms of effectiveness, we present several case studies of real corporate social networks.

This paper proposes an effective approach to provide relevant search terms for conceptual Web search. 'Semantic Term Suggestion' function has been included so that users can find the most appropriate query term to what they really need. Conventional approaches for term suggestion involve extracting frequently occurring key terms from retrieved documents. They must deal with term extraction difficulties and interference from irrelevant documents. In this paper, we propose a semantic term suggestion function called Collective Intelligence based Term Suggestion (CITS). CITS provides a novel social-network based framework for relevant terms suggestion with a semantic graph of the search term without limiting to the specific query term. A visualization of semantic graph is presented to the users to help browsing search results from related terms in the semantic graph. The search results are ranked each time according to their relevance to the related terms in the entire query session. Comparing to two popular commercial search engines, a user study of 18 users on 50 search terms showed better user satisfactions and indicated the potential usefulness of proposed method in real-world search applications.

The six papers in this special section on communities and media computing cover three broad areas: content analysis; application design; and system optimization.

We formalize data scaling classification (DSC) as a technique to trade the accuracy of classification with the network transmission load in stream analysis frameworks. We apply the proposed data scaling approaches to ECG classification in remote health monitoring systems. Experimental results show satisfactory resource savings for small amounts of utility degradation (e.g., 33% of bandwidth saving for a 3.3% of accuracy degradation).

In this paper, we propose a Semantic Graphs for Image Search (SGIS) system, which provides a novel way for image search by utilizing collaborative knowledge in Wikipedia and network analysis to form semantic graphs for search-term suggestion. The collaborative article editing process of Wikipediapsilas contributors is formalized as bipartite graphs that are folded into networks between terms. When user types in a search term, SGIS automatically retrieves an interactive semantic graph of related terms that allow users easily find related images not limited to a specific search term. Interactive semantic graph then serves as an interface to retrieve images through existing commercial search engines. This method significantly saves userspsila time by avoiding multiple search keywords that are usually required in generic search engines. It benefits both naive user who does not possess a large vocabulary (e.g., students) and professionals who look for images on a regular basis. In our experiments, 85% of the participants favored SGIS system than commercial search engines.

In this paper, a binary hierarchical classifier with feature selection is proposed for multi-class audio event classification for healthcare applications. We consider the hierarchical clustering and the feature selection problems jointly when building a binary hierarchical classifier. The proposed method results in the classifier structure as well as a compact feature subset for each component classifier for constructing the overall binary hierarchical classifier. With Support Vector Machine (SVM) for the component classifiers in our experiment, results from classifying several key audio events for the eldercare application show competitive performance to the traditional one-against-one method while the number of training and testing SVM is less in our proposed scheme. Moreover, feature selection facilitates the training of the component classifier by filtering out possible redundant and irrelevant feature components.

In this paper, we propose a Collaborative Knowledge Semantic Graphs Image Search (CKSGIS) system. It provides a novel way to conduct image search by utilizing the collaborative nature in Wikipedia and by performing network analysis to form semantic graphs for search-term expansion. The collaborative article editing process used by Wikipedia's contributors is formalized as bipartite graphs that are folded into networks between terms. When a user types in a search term, CKSGIS automatically retrieves an interactive semantic graph of related terms that allow users to easily find related images not limited to a specific search term. Interactive semantic graph then serve as an interface to retrieve images through existing commercial search engines. This method significantly saves users' time by avoiding multiple search keywords that are usually required in generic search engines. It benefits both naïve users who do not possess a large vocabulary and professionals who look for images on a regular basis. In our experiments, 85% of the participants favored CKSGIS system rather than commercial search engines.

Employees depend on other people in the enterprise for rapid access to important information. But current systems for finding experts do not adequately address the social implications of finding and engaging strangers in conversation. This paper provides a user study of SmallBlue, a social-context-aware expertise search system that can be used to identify experts, see dynamic profile information and get information about the social distance to the expert, before deciding whether and how to initiate contact. The system uses an innovative approach to privacy to infer content and dynamic social networks from email and chat logs. We describe usage of SmallBlue and discuss implications for the next generation of enterprise-wide systems for finding people.

In this paper, we investigate the possibility of using simple multimodality sensors to automatically detect a person's sleep condition. Sleep latency and sleep efficiency are critical to both sleep-related diseases and sleep quality measurements. We propose a system which consists of heart-rate, video, and audio sensors, and apply machine learning methods to infer the sleep-awake condition during the time a user spends on the bed. The sleep-awake conditions will be useful information for inferring the sleep quality. Our experimental results are promising and show the potential use of the proposed novel economical alternative to the traditional medical measurement equipment, with competitive performance on the sleep-related activity monitoring and the sleep quality measurements

While most video surveillance systems have centralized architectures that transmit videos to a central location for storage or real-time interpretation by human operators, a novel development of smart video sensors can significantly reduce the transmission load by extracting feature streams of videos that are enough for automatic semantic understanding. The overall video recognition processes are distributed into front-end sensors and back-end classifiers. Such distributed processing mechanism would significantly alleviate mundane or time-critical activities performed by human operators, and provide better network scalability. In this paper, we describe our implementation of smart video semantic sensors, which capture the context feature of the environments of the user. These sensors are used for recognizing the scenes, objects and events of the environment

Many emerging applications such as video sensor monitoring can benefit from an on-line video correlation system, which can be used to discover linkages between different video streams in realtime. However, on-line video correlations are often resource-intensive where a single host can be easily overloaded. We present a novel adaptive distributed on-line video correlation system called ViCo. Unlike single stream processing, correlations between different video streams require a distributed execution system to observe a new correlation constraint that any two correlated data must be distributed to the same host. ViCo achieves three unique features: (1) correlation-awareness that ViCo can guarantee the correlation accuracy while spreading excessive workload on multiple hosts; (2) adaptability that the system can adjust algorithm behaviors and switch between different algorithms to adapt to dynamic stream environments; and (3) fine-granularity that the workload of one resource-intensive correlation request can be divided and distributed among multiple hosts. We have implemented and deployed a prototype of ViCo on a commercial cluster system. Our experiment results using both real videos and synthetic workloads show that ViCo outperforms existing techniques for scaling-up the performance of video correlations.

We exploit dynamic patterns from both documents' and users' aspects to build models for recommendation. We propose a Community-Based Dynamic Recommendation (CBDR) scheme to make recommendations by taking content semantics, evolutionary patterns, and user communities into consideration. A Time-Sensitive Adaboost algorithm is proposed to build adaptive user models for ranking document candidates based on leveraging dynamic factors such as freshness, popularity, and other attributes. Our experimental results on a large online application system demonstrate the recommendation usefulness of the CBDR scheme is 259% better than the collaborative filtering, 126% better than the community-based static recommendation algorithm, and 106% better than the optimal global recommendation bound.

We propose that the information access behavior of a group of people can be modeled as an information flow issue, in which people intentionally or unintentionally influence and inspire each other, thus creating an interest in retrieving or getting a specific kind of information or product. Information flow models how information is propagated in a social network. It can be a real social network where interactions between people reside; it can be, moreover, a virtual social network in that people only influence each other unintentionally, for instance, through collaborative filtering. We leverage users' access patterns to model information flow and generate effective personalized recommendations. First, an early adoption based information flow (EABIF) network describes the influential relationships between people. Second, based on the fact that adoption is typically category specific, we propose a topic-sensitive EABIF (TEABIF) network, in which access patterns are clustered with respect to the categories. Once an item has been accessed by early adopters, personalized recommendations are achieved by estimating whom the information will be propagated to with high probabilities. In our experiments with an online document recommendation system, the results demonstrate that the EABIF and the TEABIF can respectively achieve an improved (precision, recall) of (91.0%, 87.1%) and (108.5%, 112.8%) compared to traditional collaborative filtering, given an early adopter exists.

We propose a layered dynamic mixture model for asynchronous multi-modal fusion for unsupervised pattern discovery in video. The lower layer of the model uses generative temporal structures such as a hierarchical hidden Markov model to convert the audiovisual streams into mid-level labels, it also models the correlations in text with probabilistic latent semantic analysis. The upper layer fuses the statistical evidence across diverse modalities with a flexible meta-mixture model that assumes loose temporal correspondence. Evaluation on a large news database shows that multi-modal clusters have better correspondence to news topics than audio-visual clusters alone; novel analysis techniques suggest that meaningful clusters occur when the prediction of salient features by the model concurs with those shown in the story clusters.

In this paper, we propose a semantic routing and filtering framework for large-scale monitoring of video streams. Our goal is to build a distributed system that at any given time is capable of simultaneously monitoring the content of multiple video streams being transmitted over the Internet (or a proprietary network). A key design requirement of such a system is the ability to handle tens of gigabytes of multimedia data per second. Traditional techniques have an important limitation. Once a bottleneck in terms of CPU or storage is reached, data is dropped indiscriminately.. In this paper, we propose distributed real-time semantic filters to route and filter video data. We propose a mechanism to alter the accuracy of classification with the complexity of execution; thus avoiding system failure during periods of overload. We propose a set of novel video features that perform better than our previous semantic classifiers. This system is capable of classifying over a hundred concepts. Experiments on 190 hours of pre-stored and live video streams validate the effectiveness of the proposed system.

Modeling visual concepts using supervised or unsupervised machine learning approaches are becoming increasing important for video semantic indexing, retrieval, and filtering applications. Naturally, videos include multimodality data such as audio, speech, visual and text, which are combined to infer therein the overall semantic concepts. However, in the literature, most researches were conducted within only one single domain. In this paper we propose an unsupervised technique that builds context-independent keyword lists for desired visual concept modeling using WordNet. Furthermore, we propose an Extended Speech-based Visual Concept (ESVC) model to reorder and extend the above keyword lists by supervised learning based on multimodality annotation. Experimental results show that the context-independent models can achieve comparable performance compared to conventional supervised learning algorithms, and the ESVC model achieves about 53% and 28.4% improvement in two testing subsets of the TRECVID 2003 corpus over a state-of-the-art speech-based video concept detection algorithm.

In this paper, we propose a new way to automatically model and predict human behavior of receiving and disseminating information by analyzing the contact and content of personal communications. A personal profile, called CommunityNet, is established for each individual based on a novel algorithm incorporating contact, content, and time information simultaneously. It can be used for personal social capital management. Clusters of CommunityNets provide a view of informal networks for organization management. Our new algorithm is developed based on the combination of dynamic algorithms in the social network field and the semantic content classification methods in the natural language processing and machine learning literatures. We tested CommunityNets on the Enron Email corpus and report experimental results including filtering, prediction, and recommendation capabilities. We show that the personal behavior and intention are somewhat predictable based on these models. For instance, "to whom a person is going to send a specific email" can be predicted by one's personal social network and content analysis. Experimental results show the prediction accuracy of the proposed adaptive algorithm is 58% better than the social network-based predictions, and is 75% better than an aggregated model based on Latent Dirichlet Allocation with social network enhancement. Two online demo systems we developed that allow interactive exploration of CommunityNet are also discussed.

This paper describes multimodal systems for ad-hoc search constructed by IBM for the TRECVID 2003 benchmark of search systems for broadcast video. These systems all use a late fusion of independently developed speech-based and visual content-based retrieval systems and outperform our individual retrieval systems on both manual and interactive search tasks. For the manual task, our best system used a query-dependent linear weighting between speech-based and image-based retrieval systems. This system has mean average precision (MAP) performance 20% above our best unimodal system for manual search. For the interactive task, where the user has full knowledge of the query topic and the performance of the individual search systems, our best system used an interlacing approach. The user determines the (subjectively) optimal weights A and B for the speech-based and image-based systems, where the multimodal result set is aggregated by combining the top A documents from system A followed by top B documents of system B and then repeating this process until the desired result set size is achieved. This multimodal interactive search has MAP 40% above our best unimodal interactive search system.

We present our new results in news video story segmentation and classification in the context of the TRECVID video retrieval benchmarking event 2003. We applied and extended the maximum entropy statistical model to fuse diverse features effectively from multiple levels and modalities, including visual, audio, and text. We have included various features such as motion, face, music/speech types, prosody, and high-level text segmentation information. The statistical fusion model is used to discover automatically relevant features contributing to the detection of story boundaries. One novel aspect of our method is the use of a feature wrapper to address different types of features - asynchronous, discrete, continuous and delta ones. We also developed several novel features related to prosody. Using the large news video set from the TRECVID 2003 benchmark, we demonstrate satisfactory performance (F1 measure up to 0.76) and, more importantly, observe an interesting opportunity for further improvement.

Video media carries multimodal information including visual, audio, textual data. Considerable research has been focused on utilizing multimodal features for better understanding of video content. However, many problems remain such as how to combine multimodal features and what are the effects of different combinations. In this paper, we propose to find the optimal combination of multimodal information in order to improve the performance of video concept detection using two methods, one is gradient-descent-optimization linear fusion and the other is super-kernel nonlinear fusion. Gradient-descent-optimization linear fusion learns an optimal weighted linear combination of single modalities based on fusing individual kernel matrices with gradient descent techniques. Super-kernel nonlinear fusion trains separate classifiers for single modalities as the first step. Once individual models have been designed, super-kernel nonlinear fusion learns an optimal nonlinear combination of individual models by fusing single-modality classifiers. Our experiments show that both methods improve performance significantly on TREC-Video 2003 benchmarks.

The work presents the first effort to automatically annotate the semantic meanings of temporal video patterns obtained through unsupervised discovery processes. This problem is interesting in domains where neither perceptual patterns nor semantic concepts have simple structures. The patterns in video are modeled with hierarchical hidden Markov models (HHMM), with efficient algorithms to learn the parameters, the model complexity and the relevant features; the meanings are contained in words of the speech transcript of the video. The pattern-word association is obtained via cooccurrence analysis and statistical machine translation models. Promising results are obtained through extensive experiments on 20+ hours of TRECVID news videos: video patterns that associate with distinct topics such as el-nino and politics are identified; the HHMM temporal structure model compares favorably to a nontemporal clustering algorithm.

We describe a novel content transcoding middleware for accessing military geospatial intelligence information in real-time. Intelligence information, including maps and location, category and properties of object targets, is adapted for various pervasive devices such as laptop, personal digital assistant (PDA), cellular phone, etc. The middleware is deployed as proxies on the Web using the IBM Websphere Transcoding Publisher (WTP) platform, which facilitates the middleware management. We developed several Java-based plug-ins and Extensible Stylesheet Language (XSL) stylesheets for content transcoding. A prototype has been established and real experiments have demonstrated the effectiveness of this novel middleware.

We propose novel techniques for image/video authentication at the semantic level. These methods use statistical learning, visual object segmentation and classification schemes for semantic understanding of visual content. A public digital signature, robust to rotation, scaling, and translation, is generated. The authentication process is executed by comparing the classification result with the information carried by the digital signature. This method leads the authentication system to learn the semantic content of multimedia data and to perform the authentication task at the semantic level.

A large number of interesting visual semantic concepts occur at a sub-frame granularity in images and occupy one or more regions at the sub-frame level. Detecting these concepts is a challenge due to segmentation imperfections. We propose multi-granular detection of visual concepts that have regional support. We build a single set of support vector machine based binary concept models from the training set with manually marked up regions. In this paper, we show that detection can be significantly improved by scoring these models over multiple granularities in the test set images, where the regions are automatically detected as a preprocessing step in detection. Using 27 regional semantic concepts from the NIST TRECVID 2003 common annotation lexicon and the corpus, we demonstrate that multi-granular detection leads to improvement in detection.

With the ambient use of digital images and the increasing concern on their integrity and originality, consumers are facing an emergent need of authenticating degraded images despite lossy compression and packet loss. In this paper, we propose a scheme to meet this need by incorporating a watermarking solution into a traditional crypto signature scheme to make the digital signatures robust to image degradations. The proposed approach is compatible with traditional crypto signature schemes except that the original image needs to be watermarked in order to guarantee the robustness of its derived digital signature. We demonstrate the effectiveness of this proposed scheme through practical experimental results as well as illustrative analysis.

Data clustering is an important technique for visual data management. Most previous work focuses on clustering video data within single sources. We address the problem of clustering across sources, and propose novel spectral clustering algorithms for multisource clustering problems. Spectral clustering is a new discriminative method realizing clustering by partitioning data graphs. We represent multi-source data as bipartite or K-partite graphs, and investigate the spectral clustering algorithm under these representations. The algorithms are evaluated using the TRECVID-2003 corpus with semantic features extracted from speech transcripts and visual concept recognition results from videos. The experiments show that the proposed bipartite clustering algorithm significantly outperforms the regular spectral clustering algorithm in capturing cross-source associations.

In this paper, we describe a novel end-to-end video automatic labeling system, which accepts MPEG-I sequence inputs and generates MPEG-7 XML metadata files based on the prior established anchor models. Seven modules were developed for the system: shot segmentation, region segmentation, annotation, feature extraction, model learning, classification, and XML rendering. The performance of this system has been tested in the NIST TREC-2002 video concept detection benchmark. The proposed system performs best in the mean average precision out of 18 worldwide participants.

In this paper, we investigate a new method for video database retrieval using interactive search fusion. Recent video analysis techniques have enabled the extraction of a variety of descriptors of features, concepts, clusters, classification results, speech and textual terms, MPEG-7 metadata, and so on. However, given an information need users are faced with a daunting task of trying to formulate queries over these multiple disparate data sources in order to retrieve the desired video content. In this paper, we explore a novel approach based on search fusion in which the user interactively builds a query by sequentially choosing among the descriptors and data sources and by selecting from various combining and score aggregation functions to fuse results of individual searches. For example, the system allows building of queries such as "retrieve video clips that have color of beach scenes, the detection of sky, and detection of water". In this paper we present the search fusion method and evaluate the performance on a large video database.

In this paper, we describe our classifier fusion framework for the visual concept detections of NIST TREC-2002 video retrieval benchmark. A normalized ensemble fusion is explored to improve overall performance by incorporating normalization of confidence scores, aggregation via combiner function, and an optimize selection. The normalized classifier fusion shows significant detection improvements for our visual concepts.

Health activity monitoring (HAM) has received increasing attention due to the rapid advances of both hardware and software technologies and strong environmental and public health needs. In this paper, we describe the architecture and implementation of the Epi-SPIRE prototype, which is a novel health activity monitoring system that generates alerts from environmental, behavioral, and public health data sources. A model-based approach is used to develop disease and behavior models from multi-modal heterogeneous data sources. Furthermore, a model-based indexing technique has been developed to speed up the data access and retrieval. This system has been successfully applied to various genuine and simulated diseases outbreaks scenarios'.

Extraction of semantic features from visual concepts is essential for meaningful content management in terms of filtering, searching and retrieval. Recently, machine learning techniques have been shown to provide a computational framework to map low level features to high level semantics. In this paper we expose these techniques to the challenge of supporting a moderately large lexicon of semantic concepts. Using the TREC 2002 benchmark corpus for training and validation we investigate a support vector machine based learning system for modeling 34 visual concepts. The detection results show excellent performance for a set of concepts with moderately large training samples. Promising performance is also observed for concepts with few training concepts.

In this paper, classifier fusion is adopted to demonstrate improved performance for our text overlay detections in the NIST TREC-2002 video retrieval benchmark. A normalized ensemble fusion is explored to combine two text overlay detection models. The fusion incorporates normalization of confidence scores, aggregation via combiner function, and an optimize selection. The proposed fusion classifier resulted best out of 11 detectors submitted to the NIST text overlay detection benchmarking and its average precision performance is 227% of the second best detector in the benchmark.

In this demo, we show a novel end-to-end video automatic labeling system, which accepts MPEG-1 sequence inputs and generates MPEG-7 XML metadata files. Detections are based on the prior established anchor models. This system has two parts: model training process and labeling process. They are comprised of seven modules: Shot Segmentation, Region Segmentation, Annotation, Feature Extraction, Model Learning, Classification, and XML Rendering.

This paper describes progress towards a general framework for incorporating multimodal cues into a trainable system for automatically annotating user-defined semantic concepts in broadcast video. Models of arbitrary concepts are constructed by building classifiers in a score space defined by a pre-deployed set of multimodal models. Results show annotation for user-defined concepts both in and outside the pre-deployed set is competitive with our best video-only models on the TREC Video 2002 corpus. An interesting side result shows speech-only models give performance comparable to our best video-only models for detecting visual concepts such as "outdoors", "face" and "cityscape".

Supporting semantic queries is a challenging problem in video retrieval. We propose the use of a lexicon of semantic concepts for handling the queries. We also propose automatic modeling of lexicon items using probabilistic techniques. We use Gaussian mixture models to build computational representations for a variety of semantic concepts including rocket-launch, outdoor greenery, sky etc. Training requires a large amount of annotated (labeled) data. Using the TREC Video test bed we compare the performance of this system supporting query by keywords with the conventional approach of query by example. Results demonstrate significant gains in performance using the automatically learnt models of semantic concepts.

We present a new compact spatio-temporal texture descriptor designed for indexing dynamic video content. Video texture provides a way to characterize spatio-temporal features such as those corresponding to splashing water, flying birds, blowing trees, and so forth, which are not easily characterized by static feature descriptors. The video texture descriptor measures the 3D wavelet energy corresponding to spatio-temporal-frequency subbands of video segments. The wavelet energy captures the texture patterns as they unfold simultaneously along the spatial and temporal dimensions. We describe the process for extracting video texture features and investigate different methods of constructing video texture descriptors from 3D wavelet energy. We evaluate the video texture descriptors in retrieval experiments and show performance improvements compared to methods based on traditional texture descriptors.

We describe a system for content-based retrieval of video that involves a series of query interactions with the user. The proposed approach allows the user, iteratively and selectively, to integrate different feature- and model-based methods of querying in the search process. This allows the user to choose among different retrieved content, features and matching dimensions, and classifiers, as appropriate, given the query objective and interim retrieval results. We investigate several approaches for integrating featureand model-based queries and results in successive query rounds including iterative filtering, score aggregation, and relevance feedback searching. We describe experimental results of applying the interactive content-based retrieval method to an automatically indexed corpus of 11 hours of video.

An MPEG system layer compressed-domain editing technique is proposed to facilitate the delivery and integration of multiple segments of MPEG files, residing on remote databases. Various multimedia applications, including retrieval and summarization, split MPEG files into small segments along shot boundaries and store them separately. This traditional method requires extra management and storage payload, provides only fixed segmentations, and may not be play smoothly. In order to solve this problem, our MPEG system-domain editing tool directly extracts video-audio information from the original MPEG sources and combines them to generate a single MPEG file. Manipulated wholly in the system bitstream domain, this method does not require decoding, re-encoding, and re-synchronization of audio and video data. Thus, it operates in real-time and provides great flexibility. This composite MPEG file can be transmitted and displayed through general Web interfaces. The proposed method is applied to our video retrieval, video summarization, and video editing systems, and has shown its great advantages.

We approach the problem of semantic multimedia retrieval as a supervised learning problem. Defining a lexicon of a small number of interesting semantic concepts we can handle a number of semantic queries. Since the number of interesting concepts available for training is usually small we explore discriminant learning techniques. In particular, we examine the use of kernel based methods and demonstrate impressive retrieval performance using semantic concepts like rocket, outdoor, greenery, sky and face. We also show that loosely coupled multimodal events can be detected based on the late fusion of detection of related auditory and visual concepts. Using a Bayesian network for inference we show how a rocket-launch event can be detected based on the detection of a related visual concept (rocket object) and a related auditory concept (explosion/blast-off).

A real-time video surveillance is presented for traffic monitoring of vehicle volume on major highways. Determining traffic volume automatically and in real-time assists drivers to dynamically plan their trips more efficiently. Our traffic monitoring system uses the virtual line graph to facilitate the detection of vehicles, classification of vehicle types, tracking of individual vehicles, and subsequently an accurate count of the number of vehicles. The virtual line analyzer detects vehicles as they cross a virtual boundary. The goal of this traffic monitoring system is to provide a real-time and accurate vehicle counter while taking advantage of stationary Web-cams, fixed highways and lanes, and deterministic vehicle characteristics.

We derive a theoretical capacity for digital image watermarking with zero transmission errors. We present three different discrete memoryless channel model to represent the watermarking process. Given the magnitude bound of noise set by applications and the acceptable watermark magnitude determined by the just-noticeable distortion, we estimate the zero-error capacity by applying Shannon's (1948) adjacency-reducing mapping technique. The capacity we estimate here corresponds to a deterministic guarantee of zero error, different from the traditional theorem approaching zero error asymptotically.

Due to the limited processing capability, memory constraints, and the power budget of mobile clients, multimedia coders and/or decoders are often difficult to implement on wireless handheld PDAs. In this Universal Tuner project, we designed and implemented a wireless video streaming system that transcodes MPEG-1/2 videos or live TV broadcasting videos to the BW or indexed color Palm OS devices. In our system, the complexity of multimedia compression and decompression algorithms is adaptively partitioned between the encoder and decoder. A mobile client would selectively disable or reenable stages of the algorithm to adapt to the device's effective processing capability. Our variable-complexity strategy of selective disabling of modules supports graceful degradation of the complexity of multimedia coding and decoding into a mobile client's low-power mode, i.e. the clock frequency of its next-generation low power CPU has been scaled down to conserve power. We modified the structure of the standard motion-compensated DCT video codecs to implement a simplified the encoder on a PC server and the decoder on a complexity-constrained PDA viewing client.

In this project, we designed a novel image authentication system based on a our semi-fragile watermarking technique. The system, called SARI, can accept quantization-based lossy compression to a determined degree without any false alarm and can sensitively detect and locate malicious manipulations. It's the first system that has such capability in distinguishing malicious attacks from acceptable operations. Furthermore, the corrupted area can be approximately recovered by the information hidden in the other part of the contentimage. The amount of information embedded in our SARI system has nearly reached the theoretical maximum zero-error information hiding capacity of digital images. The software prototype includes two parts - the watermark embedder that's freely distributed and the authenticator that can be deployed online as a third-party service or used in the recipient side.

Image authentication verifies the originality of an image by detecting malicious manipulations. Its goal is different from that of image watermarking, which embeds into the image a signature surviving most manipulations. Most existing methods for image authentication treat all types of manipulation equally (i.e., as unacceptable). However, some practical applications demand techniques that can distinguish acceptable manipulations (e.g., compression) from malicious ones. In this paper, we present an effective technique for image authentication which can prevent malicious manipulations but allow JPEG lossy compression. The authentication signature is based on the invariance of the relationships between discrete cosine transform (DCT) coefficients at the same position in separate blocks of an image. These relationships are preserved when DCT coefficients are quantized in JPEG compression. Our proposed method can distinguish malicious manipulations from JPEG lossy compression regardless of the compression ratio or the number of compression iterations. We describe adaptive methods with probabilistic guarantee to handle distortions introduced by various acceptable manipulations such as integer rounding, image filtering, image enhancement, or scaling-recaling. We also present theoretical and experimental results to demonstrate the effectiveness of the technique.

Many electronic watermarks for still images and video content are sensitive to geometric distortions. For example, simple rotation, scaling, and/or translation (RST) of an image can prevent blind detection of a public watermark. In this paper, we propose a watermarking algorithm that is robust to RST distortions. The watermark is embedded into a one-dimensional (1-D) signal obtained by taking the Fourier transform of the image, resampling the Fourier magnitudes into log-polar coordinates, and then summing a function of those magnitudes along the log-radius axis. Rotation of the image results in a cyclical shift of the extracted signal. Scaling of the image results in amplification of the extracted signal, and translation of the image has no effect on the extracted signal. We can therefore compensate for rotation with a simple search, and compensate for scaling by using the correlation coefficient as the detection measure. False positive results on a database of 10,000 images are reported. Robustness results on a database of 2000 images are described. It is shown that the watermark is robust to rotation, scale, and translation. In addition, we describe tests examining the watermarks resistance to cropping and JPEG compression.

Iterated function systems (IFSs) have received great attention in encoding and decoding fractal images. Barnsley (1988) has shown that IFSs for image compression can achieve a very high compression ratio for a single image. However, the major drawback of such a technique is the large computation load required to both encode and decode a fractal image. We provide a novel algorithm to decode IFS codes. The main features of this algorithm are that it is very suitable for parallel implementation and has no transient behavior. Also, from the decoding process of this method we can understand the encoding procedure explicitly. One example is illustrated to demonstrate the quality of its performance.

In this correspondence, we consider wavelet transform and scale space filtering as special cases of general scale shift mapping (SSM). Then, the SSM is used as a tool for characterizing the geometrical complexity of uniform fractals. This tool can reveal the construction rule of fractals and estimate fractal dimension. Finally, we use the results obtained from scale shift mapping to estimate iterated function system (IFS) codes of a class of fractal images. Some examples have been illustrated to demonstrate its good performance.