Automatic generation of Chinese fonts that consist of large numbers of glyphs with complicated structures is now still a challenging and ongoing problem in areas of AI and Computer Graphics (CG). Traditional CG-based methods typically rely heavily on manual interventions, while recentlypopularized deep learning-based end-to-end approaches often obtain synthesis results with incorrect structures and/or serious artifacts. To address those problems, this paper proposes a structure-guided Chinese font generation system, SCFont, by using deep stacked networks. The key idea is to integrate the domain knowledge of Chinese characters with deep generative networks to ensure that high-quality glyphs with correct structures can be synthesized. More specifically, we first apply a CNN model to learn how to transfer the writing trajectories with separated strokes in the reference font style into those in the target style. Then, we train another CNN model learning how to recover shape details on the contour for synthesized writing trajectories. Experimental results validate the superiority of the proposed SCFont compared to the state of the art in both visual and quantitative assessments.

In this paper, we present a novel approach for dynamic text effects transfer by using example-based texture synthesis. In contrast to previous works that require an input video of the target to provide motion guidance, we aim to animate a still image of the target text by transferring the desired dynamic effects from an observed exemplar. Due to the simplicity of target guidance and complexity of realistic effects, it is prone to producing temporal artifacts such as flickers and pulsations. To address the problem, our core idea is to find a common Nearest-neighbor Field (NNF) that would optimize the textural coherence across all keyframes simultaneously. With the static NNF for video sequences, we implicitly transfer motion properties from source to target. We also introduce a guided NNF search by employing the distance-based weight map and Simulated Annealing (SA) for deep direction-guided propagation to allow intense dynamic effects to be completely transferred with no semantic guidance provided. Experimental results demonstrate the effectiveness and superiority of our method in dynamic text effects transfer through extensive comparisons with state-of-the-art algorithms. We also show the potentiality of our method via multiple experiments for various application domains.

Point cloud analysis has drawn broader attentions due to its increasing demands in various fields. Despite the impressive performance has been achieved on several databases, researchers neglect the fact that the orientation of those point cloud data is aligned. Varying the orientation of point cloud may lead to the degradation of performance, restricting the capacity of generalizing to real applications where the prior of orientation is often unknown. In this paper, we propose the point projection feature, which is invariant to the rotation of the input point cloud. A novel architecture is designed to mine features of different levels. We adopt a PointNet-based backbone to extract global feature for point cloud, and the graph aggregation operation to perceive local shape structure. Besides, we introduce an efficient key point descriptor to assign each point with different response and help recognize the overall geometry. Mathematical analyses and experimental results demonstrate that the proposed method can extract strictly rotation-invariant representations for point cloud recognition and segmentation without data augmentation, and outperforms other state-of-the-art methods.

Abstract Despite the recent impressive development of deep neural networks, using deep learning based methods to generate large‐scale Chinese fonts is still a rather challenging task due to the huge number of intricate Chinese glyphs, e.g., the official standard Chinese charset GB18030‐2000 consists of 27,533 Chinese characters. Until now, most existing models for this task adopt Convolutional Neural Networks (CNNs) to generate bitmap images of Chinese characters due to CNN based models' remarkable success in various applications. However, CNN based models focus more on image‐level features while usually ignore stroke order information when writing characters. Instead, we treat Chinese characters as sequences of points (i.e., writing trajectories) and propose to handle this task via an effective Recurrent Neural Network (RNN) model with monotonic attention mechanism, which can learn from as few as hundreds of training samples and then synthesize glyphs for remaining thousands of characters in the same style. Experimental results show that our proposed FontRNN can be used for synthesizing large‐scale Chinese fonts as well as generating realistic Chinese handwritings efficiently.

Automatic generation of artistic glyph images is a challenging task that attracts many research interests. Previous methods either are specifically designed for shape synthesis or focus on texture transfer. In this paper, we propose a novel model, AGIS-Net, to transfer both shape and texture styles in one-stage with only a few stylized samples. To achieve this goal, we first disentangle the representations for content and style by using two encoders, ensuring the multi-content and multi-style generation. Then we utilize two collaboratively working decoders to generate the glyph shape image and its texture image simultaneously. In addition, we introduce a local texture refinement loss to further improve the quality of the synthesized textures. In this manner, our one-stage model is much more efficient and effective than other multi-stage stacked methods. We also propose a large-scale dataset with Chinese glyph images in various shape and texture styles, rendered from 35 professional-designed artistic fonts with 7,326 characters and 2,460 synthetic artistic fonts with 639 characters, to validate the effectiveness and extendability of our method. Extensive experiments on both English and Chinese artistic glyph image datasets demonstrate the superiority of our model in generating high-quality stylized glyph images against other state-of-the-art methods.

Generating personal handwriting fonts with large amounts of characters is a boring and time-consuming task. For example, the official standard GB18030-2000 for commercial font products consists of 27,533 Chinese characters. Consistently and correctly writing out such huge amounts of characters is usually an impossible mission for ordinary people. To solve this problem, we propose a system, EasyFont , to automatically synthesize personal handwriting for all (e.g., Chinese) characters in the font library by learning style from a small number (as few as 1%) of carefully-selected samples written by an ordinary person. Major technical contributions of our system are twofold. First, we design an effective stroke extraction algorithm that constructs best-suited reference data from a trained font skeleton manifold and then establishes correspondence between target and reference characters via a non-rigid point set registration approach. Second, we develop a set of novel techniques to learn and recover users’ overall handwriting styles and detailed handwriting behaviors. Experiments including Turing tests with 97 participants demonstrate that the proposed system generates high-quality synthesis results, which are indistinguishable from original handwritings. Using our system, for the first time, the practical handwriting font library in a user’s personal style with arbitrarily large numbers of Chinese characters can be generated automatically. It can also be observed from our experiments that recently-popularized deep learning based end-to-end methods are not able to properly handle this task, which implies the necessity of expert knowledge and handcrafted rules for many applications.

In this paper, we present a general-purpose solution to interactive texture transfer problems that better preserves both local structure and visual richness. It is challenging due to the diversity of tasks and the simplicity of required user guidance. The core idea of our common framework is to use multiple custom channels to dynamically guide the synthesis process. For interactivity, users can control the spatial distribution of stylized textures via semantic channels. The structure guidance, acquired by two stages of automatic extraction and propagation of structure information, provides a prior for initialization and preserves the salient structure by searching the nearest neighbor fields (NNF) with structure coherence. Meanwhile, texture coherence is also exploited to maintain similar style with the source image. In addition, we leverage an improved PatchMatch with extended NNF and matrix operations to obtain transformable source patches with richer geometric information at high speed. We demonstrate the effectiveness and superiority of our method on a variety of scenes through extensive comparisons with state-of-the-art algorithms.

Quote is a language phenomenon of transcribing the statement of someone else, such as a proverb and a famous saying. An appropriate usage of quote usually equips the expression with more elegance and credibility. However, there are times when we are eager to stress our idea by citing a quote, while nothing relevant comes to mind. Therefore, it is exciting to have a recommender system which provides quote recommendations while we are writing. This article extends previous study of quote recommendation, the task that recommends the appropriate quote according to the context (i.e., the content occurring before and after the quote). In this article, a quote recommender system called QuoteRec is presented to tackle the task. We investigate two models to learn the vector representations of quotes and contexts, and then rank the candidate quotes based on the representations. The first model learns the quote representation according to the contexts of a quote. The second model is an extension of the neural network model in previous study, which learns the representation of a quote by concerning both its content and contexts. Experimental results demonstrate the effectiveness of the two models in learning the semantic representations of quotes, and the neural network model achieves state-of-the-art results on the quote recommendation task.

Abstractive summarization is the ultimate goal of document summarization research, but previously it is less investigated due to the immaturity of text generation techniques.Recently impressive progress has been made to abstractive sentence summarization using neural models.Unfortunately, attempts on abstractive document summarization are still in a primitive stage, and the evaluation results are worse than extractive methods on benchmark datasets.In this paper, we review the difficulties of neural abstractive document summarization, and propose a novel graph-based attention mechanism in the sequence-to-sequence framework.The intuition is to address the saliency factor of summarization, which has been overlooked by prior works.Experimental results demonstrate our model is able to achieve considerable improvement over previous neural abstractive models.The data-driven neural abstractive method is also competitive with state-of-the-art extractive methods.

Headline generation is a task of abstractive text summarization, and previously suffers from the immaturity of natural language generation techniques. Recent success of neural sentence summarization models shows the capacity of generating informative, fluent headlines conditioned on selected recapitulative sentences. In this paper, we investigate the extension of sentence summarization models to the document headline generation task. The challenge is that extending the sentence summarization model to consider more document information will mostly confuse the model and hurt the performance. In this paper, we propose a coarse-to-fine approach, which first identifies the important sentences of a document using document summarization techniques, and then exploits a multi-sentence summarization model with hierarchical attention to leverage the important sentences for headline generation. Experimental results on a large real dataset demonstrate the proposed approach significantly improves the performance of neural sentence summarization models on the headline generation task.

We study the task of constructing sports news report automatically from live commentary and focus on content selection.Rather than receiving every piece of text of a sports match before news construction, as in previous related work, we novelly verify the feasibility of a more challenging setting to generate news report on the fly by treating live text input as a stream.We design scoring functions to address different requirements of the task and use stream substitution for sentence selection.Experiments suggest that our proposed framework can already produce comparable results compared with previous work that relies on a supervised learning-to-rank model.

The task of tweet timeline generation (TTG) aims at selecting a small set of representative tweets to generate a meaningful timeline and providing enough coverage for a given topical query. This paper presents an approach based on determinantal point processes (DPPs) by jointly modeling the topical relevance of each selected tweet and overall selectional diversity. Aiming at better treatment for balancing relevance and diversity, we introduce two novel strategies, namely spectral rescaling and topical prior. Extensive experiments on the public TREC 2014 dataset demonstrate that our proposed DPP model along with the two strategies can achieve fairly competitive results against the state-of-the-art TTG systems.

Quote is a language phenomenon of transcribing the saying of someone else. Proper usage of quote can usually make the statement more elegant and convincing. However, the ability of quote usage is usually limited by the amount of quotes one remembers or knows. Quote recommendation is a task of exploiting abundant quote repositories to help people make better use of quotes while writing. The task is different from conventional recommendation tasks due to the characteristic of quote. A pilot study has explored this task by using a learning to rank framework and manually designed features. However, it is still hard to model the meaning of a quote, which is an interesting and challenging problem. In this paper, we propose a neural network approach based on LSTMs to the quote recommendation task. We directly learn the distributed meaning representations for the contexts and the quotes, and then measure the relevance based on the meaning representations. In particular, we try to represent the words in quotes with specific embeddings, according to the contexts, topics and even author preferences of the quotes. Experimental results on a large dataset show that our proposed approach achieves the state-of-the-art performance and it outperforms several strong baselines.

Sentiment analysis of microblog texts has drawn lots of attention in both the academic and industrial fields. However, most of the current work only focuses on polarity classification. In this paper, we present an opinion mining system for Chinese microblogs called CMiner. Instead of polarity classification, CMiner focuses on more complicated opinion mining tasks - opinion target extraction and opinion summarization. Novel algorithms are developed for the two tasks and integrated into the end-to-end system. CMiner can help to effectively understand the users' opinion towards different opinion targets in a microblog topic. Specially, we develop an unsupervised label propagation algorithm for opinion target extraction. The opinion targets of all messages in a topic are collectively extracted based on the assumption that similar messages may focus on similar opinion targets. In addition, we build an aspect-based opinion summarization framework for microblog topics. After getting the opinion targets of all the microblog messages in a topic, we cluster the opinion targets into several groups and extract representative targets and summaries for each group. A co-ranking algorithm is proposed to rank both the opinion targets and microblog sentences simultaneously. Experimental results on a benchmark dataset show the effectiveness of our system and the algorithms.

In this paper, we propose and address a novel task of recommending quotes for writing. Quote is short for quotation, which is the repetition of someone else’s statement or thoughts. It is a common case in our writing when we would like to cite someone’s statement, like a proverb or a statement by some famous people, to make our composition more elegant or convincing. However, sometimes we are so eager to make a citation of quote somewhere, but have no idea about the relevant quote to express our idea. Because knowing or remembering so many quotes is not easy, it is exciting to have a system to recommend relevant quotes for us while writing. In this paper we tackle this appealing AI task, and build up a learning framework for quote recommendation. We collect abundant quotes from the Internet, and mine real contexts containing these quotes from large amount of electronic books, to build up a dataset for experiments. We explore the particular features of this task, and propose a few useful features to model the characteristics of quotes and the relevance of quotes to contexts. We apply a supervised learning to rank model to integrate multiple features. Experiment results show that, our proposed approach is appropriate for this task and it outperforms other recommendation methods.

User-generated reviews are valuable resources for decision making. Identifying the aspect categories discussed in a given review sentence (e.g., “food” and “service” in restaurant reviews) is an important task of sentiment analysis and opinion mining. Given a predefined aspect category set, most previous researches leverage hand-crafted features and a classification algorithm to accomplish the task. The crucial step to achieve better performance is feature engineering which consumes much human effort and may be unstable when the product domain changes. In this paper, we propose a representation learning approach to automatically learn useful features for aspect category detection. Specifically, a semi-supervised word embedding algorithm is first proposed to obtain continuous word representations on a large set of reviews with noisy labels. Afterwards, we propose to generate deeper and hybrid features through neural networks stacked on the word vectors. A logistic regression classifier is finally trained with the hybrid features to predict the aspect category. The experiments are carried out on a benchmark dataset released by SemEval-2014. Our approach achieves the state-of-the-art performance and outperforms the best participating team as well as a few strong baselines.

The task of cross-language document summarization is to create a summary in a target language from documents in a different source language.Previous methods only involve direct extraction of automatically translated sentences from the original documents.Inspired by phrasebased machine translation, we propose a phrase-based model to simultaneously perform sentence scoring, extraction and compression.We design a greedy algorithm to approximately optimize the score function.Experimental results show that our methods outperform the state-of-theart extractive systems while maintaining similar grammatical quality.

Font recognition on a single Chinese character is a challenging task especially when the identity of the character is unknown and the number of possible font types is huge. In this paper, we propose a novel method using multi-scale sparse representation to solve the problem of large-scale font recognition on a single unknown Chinese character. Specifically, we first apply a saliency-based sampling approach, which exploits the saliency information of character contours, to segment local patches in multiple scales from salient regions. Then, corresponding local descriptors are extracted by implementing Sobel and Prewitt operators in 4 directions. After encoding the local descriptors into sparse codes, max pooling and spatial pyramid matching are employed to pool them into a sparse representation. Finally, a multi-scale sparse representation is obtained by concatenating three sparse representations which respectively correspond to three particular scales of local patches, and then the linear SVM classifier is utilized for font classification. Experiments performed on a large-scale database consisting of Chinese character images in 160 fonts show that our method achieves significantly better performance compared to the state of the art. Moreover, we also carry out experiments on a subset of the database to demonstrate the effectiveness of our saliency-based sampling approach and the proposed Sobel-Prewitt feature.

Manifold-ranking has proved to be an effective method for topic-focused multi-document summarization. As basic manifold-ranking based summarization method constructs the relationships between sentences simply by the bag-of-words cosine similarity, we believe a better similarity metric will further improve the effectiveness of manifold-ranking. In this paper, we propose a joint optimization framework, which integrates the manifold-ranking process with a similarity metric learning process. The joint framework aims at learning better sentence similarity scores and better sentence ranking scores simultaneously. Experiments on DUC datasets show the proposed joint method achieves better performance than the manifold-ranking baselines and several popular methods.

Opinion target extraction is a subtask of opinion mining which is very useful in many applications. The problem has usually been solved by training a sequence labeler on manually labeled data. However, the labeled training datasets are imbalanced in different languages, and the lack of labeled corpus in a language limits the research progress on opinion target extraction in this language. In order to address the above problem, we propose a novel system called CLOpinionMiner which investigates leveraging the rich labeled data in a source language for opinion target extraction in a different target language. In this study, we focus on English-to-Chinese cross-language opinion target extraction. Based on the English dataset, our method produces two Chinese training datasets with different features. Two labeling models for Chinese opinion target extraction are trained based on Conditional Random Fields (CRF). After that, we use a monolingual co-training algorithm to improve the performance of both models by leveraging the enormous unlabeled Chinese review texts on the web. Experimental results show the effectiveness of our proposed approach.

Nowadays images on social networking websites (e.g., Flickr) are mostly accompanied with user-contributed tags, which help cast a new light on the conventional content-based image analysis tasks such as image classification and retrieval. In order to establish a scalable social image analysis system, two issues need to be considered: 1) Supervised learning is a futile task in modeling the enormous number of concepts in the world, whereas unsupervised approaches overcome this hurdle; 2) Algorithms are required to be both spatially and temporally efficient to handle large-scale datasets. In this paper, we propose a cross-view feature learning (CVFL) framework to handle the problem of social image analysis effectively and efficiently. Through explicitly modeling the relevance between image content and tags (which is empirically shown to be visually and semantically meaningful), CVFL yields more promising results than existing methods in the experiments. More importantly, being general and descriptive, CVFL and its variants can be readily applied to other large-scale multi-view tasks in unsupervised setting.

We investigate weakly-supervised image parsing, i.e., assigning class labels to image regions by using image-level labels only. Existing studies pay main attention to the formulation of the weakly-supervised learning problem, i.e., how to propagate class labels from images to regions given an affinity graph of regions. Notably, however, the affinity graph of regions, which is generally constructed in relatively simpler settings in existing methods, is of crucial importance to the parsing performance due to the fact that the weakly-supervised parsing problem cannot be solved within a single image, and that the affinity graph enables label propagation among multiple images. In order to embed more semantics into the affinity graph, we propose novel criteria by exploiting the weak supervision information carefully, and develop two graphs: L1 semantic graph and k-NN semantic graph. Experimental results demonstrate that the proposed semantic graphs not only capture more semantic relevance, but also perform significantly better than conventional graphs in image parsing.

This paper proposes FlexiFont, a system designed to generate personal font libraries from the camera-captured character images. Compared with existing methods, our system is able to process most kinds of languages and the generated font libraries can be extended by adding new characters based on the user's requirement. Moreover, digital cameras instead of scanners are chosen as the input devices, so that it is more convenient for common people to use the system. First of all, the users should choose a default template or define their own templates, then write the characters on the printed templates according to the certain instructions. After the users upload the photos of the templates with written characters, the system will automatically correct the perspective and split the whole photo into a set of individual character images. As the final step, FlexiFont will denoise, vectorize, and normalize each character image before storing it into a TrueType file. Experimental results demonstrate the robustness and efficiency of our system.

In this paper, we provide a new method for decoding tree transduction based sentence compression models augmented with lan-guage model scores, by jointly decoding two components. In our proposed so-lution, rich local discriminative features can be easily integrated without increasing computational complexity. Utilizing an unobvious fact that the resulted two com-ponents can be independently decoded, we conduct efficient joint decoding based on dual decomposition. Experimental results show that our method outperforms tradi-tional beam search decoding and achieves the state-of-the-art performance. 1

This paper proposes a non-rigid point set registration method called Structure-Guided Coherent Point Drift (SGCPD). The key idea of our method is to utilize structural information and combine the global and local point registrations together to improve the original Coherent Point Drift (CPD) algorithm. Specifically, given two point sets, we first align them using the CPD method with Localized Operator (CPDLO). Then we divide the target point set into several subsets and apply CPDLO to each subset. Finally, we implement the above two procedures until convergence. In this manner, more detailed information can be well exploited and thus higher registration accuracy can be achieved. Experimental results demonstrate that our method outperforms the original CPD approach on both point registration accuracy and skeleton decomposition accuracy for Chinese characters.

In this paper, we address the problem of weakly-supervised image parsing, whose aim is to automatically determine the class labels of image regions given image-level labels only. In the literature, existing studies pay main attention to the formulation of the weakly-supervised learning problem, i.e., how to propagate class labels from images to regions given an affinity graph of regions. Notably, however, the affinity graph of regions, which is generally constructed in relatively simpler settings in existing methods, is of crucial importance to the parsing performance due to the fact that the weakly-supervised image parsing problem cannot be handled within a single image, and that the affinity graph facilitates label propagation among multiple images. Therefore, in contrast to existing methods, we focus on how to make the affinity graph more descriptive through embedding more semantics into it. We develop two novel graphs by leveraging the weak supervision information carefully: 1) Semantic graph, which is established upon a conventional graph by utilizing the proposed weakly-supervised criteria; 2) Semantic hypergraph, which explores both intra-image and inter-image high-order semantic relevance. Experimental results on two standard datasets demonstrate that the proposed semantic graphs and hypergraphs not only capture more semantic relevance, but also perform significantly better than conventional graphs in image parsing. More remarkably, due to the complementariness among the proposed semantic graphs and hypergraphs, the combination of them shows even more promising results.

How to automatically generate compact and high-quality vectorization for Chinese calligraphy images is a challenging problem, since these images usually suffer from noisy contours and discontinuous strokes. In this paper, we propose a skeleton guided approach to vectorize Chinese calligraphy images. Since the skeleton reflects the writing trace and it is less influenced by contour noises, our method could extract the important writing style from the noisy contours. Specifically, in our method, the calligraphy image is first preprocessed by binarization and denoising. Then salient contour points are detected by a novel algorithm. Afterwards, under the guidance of skeleton information, the salient points are classified into corner points and joint points. Finally, a dynamic curve fitting procedure is applied to generate the vectorization result. Experimental results demonstrate that our skeleton-guided approach could automatically distinguish tiny features from contour noises and thus obtains more visually satisfactory performance compared to other existing methods.

Cross-media retrieval has become a key problem in both research and application, in which users can search results across all of the media types (text, image, audio, video, and 3-D) by submitting a query of any media type. How to measure the content similarity among different media is the key challenge. Existing cross-media retrieval methods usually focus on modeling the pairwise correlation or semantic information separately. In fact, these two kinds of information are complementary to each other and optimizing them simultaneously can further improve the accuracy. In this paper, we propose a novel feature learning algorithm for cross-media data, called joint representation learning (JRL), which is able to explore jointly the correlation and semantic information in a unified optimization framework. JRL integrates the sparse and semisupervised regularization for different media types into one unified optimization problem, while existing feature learning methods generally focus on a single media type. On one hand, JRL learns sparse projection matrix for different media simultaneously, so different media can align with each other, which is robust to the noise. On the other hand, both the labeled data and unlabeled data of different media types are explored. Unlabeled examples of different media types increase the diversity of training data and boost the performance of joint representation learning. Furthermore, JRL can not only reduce the dimension of the original features, but also incorporate the cross-media correlation into the final representation, which further improves the performance of both cross-media retrieval and single-media retrieval. Experiments on two datasets with up to five media types show the effectiveness of our proposed approach, as compared with the state-of-the-art methods.

As the major component of big data, unstructured heterogeneous multimedia content such as text, image, audio, video and 3D increasing rapidly on the Internet. User demand a new type of cross-media retrieval where user can search results across various media by submitting query of any media. Since the query and the retrieved results can be of different media, how to learn a heterogeneous metric is the key challenge. Most existing metric learning algorithms only focus on a single media where all of the media objects share the same data representation. In this paper, we propose a joint graph regularized heterogeneous metric learning (JGRHML) algorithm, which integrates the structure of different media into a joint graph regularization. In JGRHML, different media are complementary to each other and optimizing them simultaneously can make the solution smoother for both media and further improve the accuracy of the final metric. Based on the heterogeneous metric, we further learn a high-level semantic metric through label propagation. JGRHML is effective to explore the semantic relationship hidden across different modalities. The experimental results on two datasets with up to five media types show the effectiveness of our proposed approach.

Microblog messages pose severe challenges for current sentiment analysis techniques due to some inherent characteristics such as the length limit and informal writing style.In this paper, we study the problem of extracting opinion targets of Chinese microblog messages.Such fine-grained word-level task has not been well investigated in microblogs yet.We propose an unsupervised label propagation algorithm to address the problem.The opinion targets of all messages in a topic are collectively extracted based on the assumption that similar messages may focus on similar opinion targets.Topics in microblogs are identified by hashtags or using clustering algorithms.Experimental results on Chinese microblogs show the effectiveness of our framework and algorithms.

Automatically establishing correspondence between Chinese characters is a challenging task. In this paper, we propose a novel method to solve this problem. Given two Chinese characters, we first extract and properly prune the skeleton of each character to get the key points and the connectivity relations of these points. Then, the similarity between each pair of key points is calculated via the comparison of their local features. Afterwards, a set of edges are constructed by considering both the connectivity relations and k nearest neighbors (k-nn) of each point. Finally, correspondence between two characters is established by applying a guided graph matching algorithm. Experimental results demonstrate the effectiveness of our method for the correspondence problem of Chinese characters in both printing and handwritten styles. Moreover, we also show that our method can be utilized to automatically extract strokes from Chinese characters.

We investigate an image classification task where the training images come along with tags, but only a subset being labeled, and the goal is to predict the class label of test images without tags. This task is crucial for image search engine on photo sharing Web sites. In previous work, it is handled by first learning a multiple kernel learning classifier using both image content and tags to score unlabeled training images, and then building up a least-squares regression (LSR) model on visual features to predict the label of test images. However, there exist three important issues in the task: (1) Image tags on photo sharing Web sites tend to be inaccurate and incomplete, and thus refining them is beneficial; (2) Supervised learning with a limited number of labeled samples may be unreliable to some extent, while a graph-based semi-supervised approach can be adopted by also considering similarities of unlabeled data; (3) LSR is established upon centered visual kernel columns and breaks the symmetry of kernel matrix, whereas support vector regression can readily use the original visual kernel and thus leverage its full power. To handle the task more effectively, we propose to combine tag refinement, graph-based learning and support vector regression together. Experimental results on the PASCAL VOC'07 and MIR Flickr datasets show the superior performance of the proposed approach.

We propose a new method to refine the result of video annotation by exploiting the semantic and visual context of video. On one hand, semantic context mining is performed in a supervised way, using the manual concept labels of the training set. It is very useful for boosting video annotation performance, because semantic context is learned from labels given by people, indicating human intention. In this paper, we model the spatial and temporal context in video by using conditional random fields with different structures. Comparing with existing methods, our method could more accurately capture concept relationship in video and could more effectively improve the video annotation performance. On the other hand, visual context mining is performed in a semi-supervised way based on the visual similarities among video shots. It indicates the natural visual property of video, and could be considered as the compensation to semantic context, which generally could not be perfectly modeled. In this paper, we construct a graph based on the visual similarities among shots. Then a semi-supervised learning approach is adopt based on the graph to propagate probabilities of the reliable shots to others having similar visual features with them. Extensive experimental results on the widely used TRECVID datasets exhibit the effectiveness of our method for improving video annotation accuracy.

We consider the problem of cross-media retrieval, where the query and the retrieved results can be of different modalities. In this paper, we propose a novel cross-modality correlation propagation approach to simultaneously deal with positive correlation and negative correlation between media objects of different modalities, while existing works focus solely on the positive correlation. Negative correlation is very important because it provides the effective exclusive information. The correlation is modeled as must-link constraints and cannot-link constraints respectively. Furthermore, our approach is able to propagate the correlation between heterogeneous modalities. Experiments on the wikipedia dataset show the effectiveness of our cross-modality correlation propagation approach, compared with state-of-the-art methods.

Opinion target extraction is a subtask of opinion mining which is very useful in many applications. In this study, we investigate the problem in a cross-language scenario which leverages the rich labeled data in a source language for opinion target extraction in a different target language. The English labeled corpus is used as training set. We generate two Chinese training datasets with different features. Two labeling models for Chinese opinion target extraction are learned based on Conditional Random Fields (CRF). After that, we use a monolingual co-training algorithm to improve the performance of both models by leveraging the enormous unlabeled Chinese review texts on the web. Experimental results show the effectiveness of our proposed approach.

In this paper, we propose the Patch-Descriptor-Similarity Space (PDSS) for unconstrained face verification, which is challenging due to image variations in pose, lighting, facial expression, and occlusion. Our proposed PDSS considers jointly patch, descriptor and similarity measure, which are ignored by the existing work. PDSS is extremely effective for face verification because each axis of PDSS will boost each other and could maximize the effect of every axis. Each point in PDSS reflects a distinct partial-matching between two facial images, which could be robust to variations in the facial images. Moreover, by selecting the discriminating point subset from PDSS, we could describe accurately the characteristic similarities and differences between two facial images, and further decide whether they represent the same person. In PDSS, each axis can describe effectively the distinct features of the faces: each patch (the first axis) reflects a distinct trait of a face; the descriptor (the second axis) is used to describe such face trait; and the similarity between two features can be measured by a certain kind of similarity measure (the third axis). The experiment adopts the extensively-used Labeled Face in the Wild (LFW) unconstrained face recognition dataset (13K faces), and our proposed PDSS approach achieves the best result, compared with the state-of-the-art methods.

We propose a new method to boost the performance of video annotation by exploiting concept relationship in temporal context. The motivation of our idea mainly comes from the fact that temporally continuous shots in video are generally with consistent content, so that concepts in these shots should be semantically relevant. We utilize a temporal model to describe the contributions of relevant concepts to the presence of a target concept. By connecting our model with conditional random field and adopting the learning and inference approaches of it, we could obtain the refined probability of a concept occurring in the shot, which is the leverage of temporal context and initial output of video annotation. Experimental results on the widely used TRECVID dataset exhibit the effectiveness of our method for improving video annotation accuracy.

There usually exist many news articles written in different languages about a hot news event. The news articles in different languages are written in different ways to reflect different standpoints. For example, the Chinese news agencies and the Western news agencies have published many articles to report the same news of Liu Xiaobo's Nobel Prize in Chinese and English languages, respectively. The Chinese news articles and the English news articles share something about the news fact in common, but they focus on different aspects in order to reflect different standpoints about the event. In this paper, we investigate the task of multilingual news summarization for the purpose of finding and summarizing the major differences between the news articles about the same event in the Chinese and English languages. We propose a novel constrained co-ranking (C-CoRank) method for addressing this special task. The C-CoRank method adds the constraints between the difference score and the common score of each sentence to the co-ranking process. Evaluation results on the manually labeled test set with 15 news topics show the effectiveness of our proposed method, and the constrained co-ranking method can outperform a few baselines and the typical co-ranking method.

This paper proposes a new approach to refine video annotation by exploiting the inter-shot context. Our method is mainly novel in two ways. On one hand, to refine annotation result of the target concept, we model the sequence of shots in video as a conditional random field with chain structure. In this way, we can capture different kinds of concept relationships in inter-shot context to improve the annotation accuracy. On the other hand, to exploit inter-shot context for the target concept, we classify shots into different types according to their correlation to the target concept, which will be used to represent different kinds of concept relationships in inter-shot context. Experiments on the widely used TRECVID 2006 data set show that our method is effective for refining video annotation, achieving a significant performance improvement over several state of the art methods.

In this paper we investigate a novel and important problem in multi-document summarization, i.e., how to extract an easy-to-understand English summary for non-native readers. Existing summarization systems extract the same kind of English summaries from English news documents for both native and non-native readers. However, the non-native readers have different English reading skills because they have different English education and learning backgrounds. An English summary which can be easily understood by native readers may be hardly understood by non-native readers. We propose to add the dimension of reading easiness or difficulty to multi-document summarization, and the proposed EUSUM system can produce easy-to-understand summaries according to the English reading skills of the readers. The sentence-level reading easiness (or difficulty) is predicted by using the SVM regression method. And the reading easiness score of each sentence is then incorporated into the summarization process. Empirical evaluation and user study have been performed and the results demonstrate that the EUSUM system can produce more easy-to-understand summaries for non-native readers than existing summarization systems, with very little sacrifice of the summary's informativeness.

Document summarization and keyphrase extraction are two related tasks in the IR and NLP fields, and both of them aim at extracting condensed representations from a single text document. Existing methods for single document summarization and keyphrase extraction usually make use of only the information contained in the specified document. This article proposes using a small number of nearest neighbor documents to improve document summarization and keyphrase extraction for the specified document, under the assumption that the neighbor documents could provide additional knowledge and more clues. The specified document is expanded to a small document set by adding a few neighbor documents close to the document, and the graph-based ranking algorithm is then applied on the expanded document set to make use of both the local information in the specified document and the global information in the neighbor documents. Experimental results on the Document Understanding Conference (DUC) benchmark datasets demonstrate the effectiveness and robustness of our proposed approaches. The cross-document sentence relationships in the expanded document set are validated to be beneficial to single document summarization, and the word cooccurrence relationships in the neighbor documents are validated to be very helpful to single document keyphrase extraction.

This paper proposes a new approach for the multiple frame integration of video, whose novelty mainly lies in three phases: Firstly, in the text-block group (TBG) identification, we identify the blocks with the same text by considering jointly the location, edge distribution and contrast of the text block. Then, in the TBG filtering, to avoid the bad effects of the blurred text on the result of integration, we measure the clarity of the text using the proposed text-intensity map, and select the blocks with the clear text for the integration.Finally, in the TBG integration, we integrate the text blocks by using the average and minimum integrations in the text and background of the image respectively,which can obtain the clean background and clear text with high contrast for the effective text recognition.The experimental results show our method can improve the performance of text recognition in video significantly.

This paper proposes a new approach for the query-by-example audio retrieval, named as segment-based manifold-ranking algorithm. Our approach adopts the audio segment, instead of the whole audio, as the basic unit for the manifold-ranking process. We formulate the query-by-example audio retrieval as a manifold-ranking problem in two stages: initial ranking and re-ranking. In the initial ranking stage, we use the existing distance functions to rank all audios according to their similarity values with the query. In the re-ranking stage, each audio is divided into some segments by the detected change points, and then the segment-based manifold-ranking algorithm is employed to re-rank the initial retrieved audios. Experimental results show the proposed approach is effective to improve the ranking capability of the existing distance functions, and the audio segment is a more appropriate unit for the manifold-ranking algorithm than the whole audio.

This paper proposes a new approach and algorithm for the semantic concept annotation based on audio PLSA (probabilistic latent semantic analysis) model. The novelty of our approach includes two sides: Audio vocabulary construction, and audio PLSA model. In audio vocabulary construction, we first segment an audio-clip into a few homogeneous audio-segments according to its content change, which not only capture the change property of audio-clip, but also keep and present the change relation and temporal order of audio features. Then an audio vocabulary is constructed by the RPCL (rival penalized competitive learning) clustering of audio-segments. In this way, each audio-clip can be represented by a bag-of-word form. In audio PLSA model, PLSA is employed to discover the latent topics existing in audio-clips. Based on the discovered topics, the concept classification is then carried out by a support vector machine (SVM) classifier. In addition, we also combine the local features extracted by PLSA and global features in audio-clip to further improve the performance of concept annotation. The experiments are evaluated on 85 hours of audio data from the TRECVID 2005, and show the encouraging results of our approach.

When fitting finite mixtures to multivariate data, it is crucial to select the appropriate number of components. Under regularization theory, we aim to resolve this “unsupervised” learning problem via regularizing the likelihood by the full entropy of posterior probabilities for finite mixture fitting. Two deterministic annealing implementations are further proposed for this entropy regularized likelihood (ERL) learning. Through some asymptotic analysis of the deterministic annealing ERL (DAERL) learning, we find that the global minimization of the ERL function in an annealing way can lead to automatic model selection on finite mixtures and also make our DAERL algorithms less sensitive to initialization than the standard EM algorithm. The simulation experiments then demonstrate that our algorithms can provide some promising results just as our theoretic analysis. Moreover, our algorithms are evaluated in the application of unsupervised image segmentation and shown to outperform other state-of-the-art methods.

This paper proposes a new approach for the text detection and extraction in image. The novelty of our approach mainly lies in the color-based clustering into two phases: In text detection phase, we consider jointly the two significant features of text regions in image: homogeneous color and sharp edges, and color-based clustering is employed to decompose the color edge map of image into several edge maps, which makes the text detection of image more accurate. In text extraction phase, on one hand, for effective text recognition, we consider the color difference between the text and background in image, and color-based clustering is utilized to remove image noise. Another hand, for effective binarization of text region, instead of performing binarization in a constant color plane as in the existing methods, our approach can adaptively select the best color plane according to the text contrast difference among color planes for binarization. Experimental results show our approach is better than the existing methods.

This paper proposes a new approach for the similarity measure and ranking of audio clips by graph modeling and matching. Instead of using frame-based or salient-based features to measure the acoustical similarity of audio clips, segment-based similarity is proposed. The novelty of our approach lies in two aspects: segment-based representation, and the similarity measure and ranking based on four kinds of similarity factors. In segmentbased representation, segments not only capture the change property of audio clip, but also keep and present the change relation and temporal order of audio features. In the similarity measure and ranking, four kinds of similarity factors: acoustical, granularity, temporal order and interference are progressively and jointly measured by optimal matching and dynamic programming, which guarantee the comprehensive and sufficient similarity measure between two audio clips. The experimental result shows that the proposed approach is better than some existing methods in terms of retrieval and ranking capabilities.

Graph-ranking based methods have been developed for generic multi-document summarization in recent years and they make uniform use of the relationships between sentences to extract salient sentences. This paper proposes to integrate the relevance of the sentences to the specified topic into the graph-ranking based method for topic-focused multi-document summarization. The cross-document relationships and the within-document relationships between sentences are differentiated and we apply the graph-ranking based method using each individual kind of sentence relationships and explore their relative importance for topic-focused multi-document summarization. Experimental results on DUC2003 and DUC2005 demonstrate the great importance of the cross-document relationships between sentences for topic-focused multi-document summarization. Even the approach based only on the cross-document sentence relationships can perform better than or at least as well as the approaches based on both kinds of sentence relationships

In this paper, a novel approach for automatic matching, ranking and retrieval of video clips is proposed. Motivated by the maximal and optimal matching theories in graph analysis, a new similarity measure of video clips is defined based on the representation and modeling of bipartite graph. Four different factors: visual similarity, granularity, interference and temporal order of shots are taken into consideration for similarity ranking. These factors are progressively analyzed in the proposed approach. Maximal matching utilizes the granularity factor to efficiently filter false matches, while optimal matching takes into account the visual, granularity and interference factors for similarity measure. Dynamic programming is also formulated to quantitatively evaluate the temporal order of shots. The final similarity measure is based on the results of optimal matching and dynamic programming. Experimental results indicate that the proposed approach is effective and efficient in retrieving and ranking similar video clips.