Knowledge-based visual question answering (KB-VQA) requires reasoning about the visual grounding relations between the images and questions by incorporating external knowledge. Existing works typically retrieve knowledge from knowledge graphs by leveraging global multimodal representations of image–text pairs for graph convolution, which neglect contextual clues at hop granularity, resulting in suboptimal spreading and leveraging of contextual information. To this end, we propose a multi-hop graph reasoning network (MGRN) for KB-VQA, which consists of a knowledge graph constructor (KGC) module, a semantic-instructed graph reasoning (SGR) module, and an answering module. MGRN exploits multimodal semantics from given images and questions as instructions for graph reasoning to obtain the knowledge representation from either the scene graph or knowledge base. Specifically, KGC fuses the scene graph with triplets from ConceptNet and Comet to construct a contextual knowledge graph for retrieving knowledge representation. Furthermore, SGR conducts multi-hop graph reasoning to select top- K knowledge items for answering by passing and filtering interplay messages on contextual knowledge graphs under the guidance of multimodal semantic representation. Extensive experiments conducted on two public datasets show the effectiveness and outperformance of our method.

Social media serves as a real-time collecting and disseminating center of users’ ideas, opinions, and experiences. The deliberate disinformation and rumors propagate rapidly online due to their exaggerated facts, controversial opinions, divisive perspectives, and stunning expressions. Rumor detection approaches typically use social media posts with rumor or non-rumor labels for training and testing without disclosing the rationale behind decision-makings. On one hand, collecting evidence data to verify claims relies on expert efforts. On the other hand, verifying the truthfulness of confusing claims with distracting and lengthy evidences is still challenging. In this paper, we contribute a rumor detection dataset with multi-granularity evidences, denoted as the RD-E dataset, which includes response, fact-check, article, sourcing data and generated evidence by large language models, supporting models to verify the truthfulness of claims on social media. A number of 32,892 claims from 4,525 public individuals and organizations are annotated to 6 kinds of labels, including true, mostly true, half true, mostly false, false, pants on fire, covering a wide range of topics, e.g., politics, economy, society, technology, and health. In the experiments, seven rumor detection models have been investigated and customized on four predefined subtasks for comparisons.

Social media allows the public to access information conveniently, in which the false messages that are eye-catching may spread fast. In this paper, we propose a two-stage confidence-guided boundary adaption (CBA) network, consisting of a feature preprocessing (FP) module, a biased ambiguity learning (BA) module and a confidence-guided boundary adaptation (CG) module. In the first stage, the FP module obtains the textual and visual features, which are fused by conducting the visual-to-textual and textual-to-visual correlation coefficients with attention mechanism. Furthermore, BA evaluates the distribution distance between fused features and single modalities to determine the weights between modalities, capturing the semantics of key modality. In the second stage, CG leverages samples from the low-confidence interval to generate new instances using a mixup of augmentation techniques, aiming to occupy the decision space and optimize the decision boundary of the classifier. Extensive experiments on two public datasets show that our CBA model is 1.6% and 2.6% higher than the state-of-the-art methods.

Visual question answering (VQA) is a challenging task that reasons over questions on images with knowledge. A prerequisite for VQA is the availability of annotated datasets, while the available datasets have several limitations. 1) The diversity of questions and answers are limited to a few question categories and certain concepts (e.g., objects, relations, actions.) with somewhat mechanical answers. 2) The availability of background knowledge or context information has been disregarded with just images, questions and answers being provided. 3) The timeliness of knowledge has not been examined, though some works may introduce factual or commonsense knowledge bases, e.g., ConceptNet, DBPedia. In this paper, we provide an Event-oriented Visual Question Answering (E-VQA) dataset including free-form questions and answers for real-world event concepts, which provides context information of events as domain knowledge in addition to images. E-VQA consists of 2,690 social media images, 9,088 questions, 5,479 answers, and 1,157 news media articles for references being annotated to 182 real-world events, covering a wide range of topics, such as armed conflicts and attacks, disasters and accidents, law and crime. For comparisons, we investigate 10 state-of-the-art VQA methods as benchmarks.

In this paper, we propose a multi-branch semantic learning network (MSLN) to generate image according to textual description by taking into account global and local textual semantics, which consists of two stages. The first stage generates a coarse-grained image based on the sentence features. In the second stage, a multi-branch fine-grained generation model is constructed to inject the sentence-level and word-level semantics into two coarse-grained images by global and local attention modules, which generate global and local fine-grained image textures, respectively. In particular, we devise a channel fusion module (CFM) to fuse the global and local fine-grained features in the multi-branch fine-grained stage and generate the output image. Extensive experiments conducted on the CUB-200 dataset and Oxford-102 dataset demonstrate the superior performance of the proposed method. (e.g., FID is reduced from 16.09 to 14.43 on CUB-200).

In this paper, a novel Deep Semantic Space learning model with Intra-class Low-rank constraint (DSSIL) is proposed for cross-modal retrieval, which is composed of two subnetworks for modality-specific representation learning, followed by projection layers for common space mapping. In particular, DSSIL takes into account semantic consistency to fuse the cross-modal data in a high-level common space, and constrains the common representation matrix within the same class to be low-rank, in order to induce the intra-class representations more relevant. More formally, two regularization terms are devised for the two aspects, which have been incorporated into the objective of DSSIL. To optimize the modality-specific subnetworks and the projection layers simultaneously by exploiting the gradient decent directly, we approximate the nonconvex low-rank constraint by minimizing a few smallest singular values of the intra-class matrix with theoretical analysis. Extensive experiments conducted on three public datasets demonstrate the competitive superiority of DSSIL for cross-modal retrieval compared with the state-of-the-art methods.

Sentiment analysis of such opinionated online texts as reviews and comments has received increasingly close attention, yet most of the work is intended to deal with the detection of authors’ emotion. In contrast, this article presents our study of the social emotion detection problem, the objective of which is to identify the evoked emotions of readers by online documents such as news articles. A novel Latent Discriminative Model (LDM) is proposed for this task. LDM works by introducing intermediate hidden variables to model the latent structure of input text corpora. To achieve this, it defines a joint distribution over emotions and latent variables, conditioned on the observed text documents. Moreover, we assume that social emotions are not independent but correlated with one another, and the dependency of them is capable of providing additional guidance to LDM in the training process. The inclusion of this emotional dependency into LDM gives rise to a new Emotional Dependency-based LDM (eLDM). We evaluate the proposed models through a series of empirical evaluations on two real-world corpora of news articles. Experimental results verify the effectiveness of LDM and eLDM in social emotion detection.

No abstract available.

The great success and high number of participants in pattern recognition related competitions last years show an important improvement of recognition and classification approaches. This success is unconceivable without the availability of huge datasets of real world data. Within the scope of the 11th International Conference on Document Analysis and Recognition (ICDAR2011) a call for competitions was initiated. The aim of the competitions is the performance evaluation of algorithms and methods for a particular task of document analysis and recognition. 22 different teams have submitted their proposals. The subjects of these propositions cover the field of research of document analysis and recognition from pre-processing over document analysis to text recognition or writer identification. 16 competitions have received enough participants (we have defined the threshold to 3 systems) to present their evaluation at the ICDAR 2011.

A novel shape descriptor based on the histogram matrix of pixel-level structural features is presented. First, length ratios and angles between the centroid and contour points of a shape are calculated as two structural attributes. Then, the attributes are combined to construct a new histogram matrix in the feature spacestatistically. The proposed shape descriptor can measure circularity, smoothness, and symmetry of shapes, and be used to recognize shapes. Experimental results demonstrate the effectiveness of our method.

A half-day single track workshop is designed to gather academic researchers and industrial practitioners at to share ideas and knowledge of know-how, and to discuss all relevant issues including the business models, enabling technologies, and killer applications, of Web-based question answering (QA), especially, the user-interactive QA services and applications. The workshop program consists of two sessions, one for academic papers, and the other for industrial practice papers. Each session consists of a leading talk, followed by three short presentations. Sufficient time is allocated for brainstorming discussions in each session.

A severe potential security problem in utilization of Unicode on the Web is identified, which is resulted from the fact that there are many similar characters in the Universal Character Set (UCS). The foundation of our solution relies on evaluating the similarity of characters in UCS. We develop a solution based on the renowned Kernel Density Estimation (KDE) method to establish such a Unicode Similarity List (UC-SimList).

An approach to detection of phishing Web pages based on visual similarity is proposed, which can be utilized as a part of an enterprise solution to antiphishing. A legitimate Web page owner can use this approach to search the Web for suspicious Web pages which are visually similar to the true Web page. The approach first decomposes the Web pages into salient (visually distinguishable) block regions. The visual similarity between two Web pages is then evaluated in three metrics: block level similarity, layout similarity, and overall style similarity. A Web page is reported as a phishing suspect if any of them (with regards to the true one) is higher than its corresponding preset threshold. Preliminary experiments show that the approach can successfully detect those phishing Web pages with few false alarms at a speed adequate for online application.

In this paper, a dynamic programming (DP) based approach is proposed to split a freehand stroke into the optimal number of line segments and elliptical arcs. Different from existing DP approaches, which often require a predefined fixed number and/or the templates of the result segments, our approach is actually a generic parameterized framework, which can be configured to some existing methods by fixing some of the parameters. Our key contributions is to avoid presetting of the number or templates of the segments and achieve the best tradeoff between the segment number and approximate error by using a suitable penalty function. Experiments show that our approach achieves high segmentation accuracy and can response to user's stroke input in real-time.

In this paper, we propose a text representation model, Tensor Space Model (TSM), which models the text by multilinear algebraic high-order tensor instead of the traditional vector. Supported by techniques of multilinear algebra, TSM offers a potent mathematical framework for analyzing the multifactor structures. TSM is further supported by certain introduced particular operations and presented tools, such as the High-Order Singular Value Decomposition (HOSVD) for dimension reduction and other applications. Experimental results on the 20 Newsgroups dataset show that TSM is constantly better than VSM for text classification.

In this paper, we propose a framework for engineering drawings recognition using a case-based approach. The key idea of our scheme is that, interactively, the user provides an example of one type of graphic object in an engineering drawing, then the system learns the graphical knowledge of this type of graphic object from the example and uses this learned knowledge to recognize or search for similar graphic objects in engineering drawings. The scheme emphasizes the following three distinct characteristics: automatism, run-time-ness, and robustness. We summarized five types of geometric constraints to represent the generic graphical knowledge. We also developed two algorithms for case-based graphical knowledge acquisition and knowledge-based graphics recognition, respectively. Experiments have shown that our proposed framework is both efficient and effective for recognizing various types of graphic objects in engineering drawings.

Line detection algorithms constitute the basis fortechnical document analysis and recognition. Theperformance of these algorithms decreases as the qualityof the documents degrades. To test the robustness of linedetection algorithms under noisy circumstance, wepropose a document degradation mode, which simulatesnoise types that drawings may undergo during theirproduction, storage, photocopying, or scanning. Using ourmodel, a series of document images at various noise levelsand types can be generated for testing the performance ofline detection algorithms. To illustrate that our model isconsistent with real world noise types, we validated themethod by applying it to three line recognition algorithms.

Thesaurus has been widely used in many applications, including information retrieval, natural language processing, and question answering. In this paper, we propose a novel approach to automatically constructing a domain-specific thesaurus from the Web using link structure information. The proposed approach is able to identify new terms and reflect the latest relationship between terms as the Web evolves. First, a set of high quality and representative websites of a specific domain is selected. After filtering out navigational links, link analysis is applied to each website to obtain its content structure. Finally, the thesaurus is constructed by merging the content structures of the selected websites. The experimental results on automatic query expansion based on our constructed thesaurus show 20% improvement in search precision compared to the baseline.

Web link analysis has been proved to significantly enhance the precision of Web searching in practice. Among existing approaches, Kleinberg's (1998) HITS and Google's PageRank are the two most representative algorithms that employ explicit hyperlink structure among Web pages to conduct link analysis, and DirectHit represents the other extreme that takes the user's access frequency as an implicit link to the Web page for assessing its importance. We propose a novel link analysis algorithm which puts both explicit and implicit link structures under a unified framework, and show that HITS and DirectHit are essentially two extreme instances of our proposed method. One important advantage of our method is its ability to analyze not only the hyperlinks between Web pages but also the interactions between users and the Web at the same time. The importance of Web pages and users can reinforce each other to improve Web link analysis. Compared with traditional HITS and DirectHit algorithms, our method further improves the search precision by 11.8% and 25.3%.

We propose an objective, comprehensive and difficulty-independent performance evaluation protocol for video text detection algorithms. The protocol includes a positive set and a negative set of indices at textbox level, which evaluate the detection quality in terms of both location accuracy and fragmentation of the detected textboxes. In the protocol, we assign a detection difficulty (DD) level to each ground truth textbox. The performance indices can then be normalized with respect to the textbox DD level and are therefore independent of the ground truth difficulty. We also assign a detection importance (DI) level to each ground truth textbox. The overall detection rate is the DI-weighted average of the detection qualities of all ground truth textboxes, which makes the detection rate more accurate to reveal the real performance. The automatic performance evaluation scheme has been applied on a text detection approach to determine the best parameters that can yield the best detection results.

An online graphics recognition system is presented, which provides users a natural, convenient, and efficient way to input rigid and regular shapes or graphic objects (e.g., triangles, rectangles, ellipses, straight line, arrowheads, etc.) by quickly drawing their sketchy shapes in single or multiple strokes. An input sketchy (hand-drawn) shape is immediately converted into the user-intended rigid shape based on the shape similarity and the time constraint of the sketchy line. Three different (rule-based, SVM-based, and ANN-based) approaches have been applied and compared in the system. Experiments and evaluation are also presented, which show good performance of the system.

No abstract available.

Abstract The popularity of digital images is rapidly increasing due to improving digital imaging technologies and convenient availability facilitated by the Internet. However, how to find user‐intended images from the Internet is nontrivial. The main reason is that the Web images are usually not annotated using semantic descriptors. In this article, we present an effective approach to and a prototype system for image retrieval from the Internet using Web mining. The system can also serve as a Web image search engine. One of the key ideas in the approach is to extract the text information on the Web pages to semantically describe the images. The text description is then combined with other low‐level image features in the image similarity assessment. Another main contribution of this work is that we apply data mining on the log of users' feedback to improve image retrieval performance in three aspects. First, the accuracy of the document space model of image representation obtained from the Web pages is improved by removing clutter and irrelevant text information. Second, to construct the user space model of users' representation of images, which is then combined with the document space model to eliminate mismatch between the page author's expression and the user's understanding and expectation. Third, to discover the relationship between low‐level and high‐level features, which is extremely useful for assigning the low‐level features' weights in similarity assessment.

Abstract A novel idea of media agent is briefly presented, which can automatically build a personalized semantic index of Web media objects for each particular user. Because the Web is a rich source of multimedia data and the text content on the Web pages is usually semantically related to those media objects on the same pages, the media agent can automatically collect the URLs and related text, and then build the index of the multimedia data, on behalf of the user whenever and wherever she accesses these multimedia data or their container Web pages. Moreover, the media agent can also use an off‐line crawler to build the index for those multimedia objects that are relevant to the user's favorites but have not accessed by the user yet. When the user wants to find these multimedia data once again, the semantic index facilitates text‐based search for her.

Accurate arc segmentation, essential for high level engineering drawing understanding is very difficult due to noise, clutter, tangency, and intersections with other geometry objects. We present an application of a generic methodology for recognition of multicomponent graphic objects in engineering drawings to the segmentation of circular arcs. The underlying mechanism is a sequential stepwise recovery of components that are segmented as wire fragments during the sparse pixel vectorization process and meet a set of continuity conditions. Proper threshold selection and consistent checking of co-circularity of the assumed arc pieces result in an accurate arc segmentation method.

A method based on connected area detection is developed to segment and recognize dimension texts in engineering drawings. First we propose an effective algorithm to find all connected areas from drawing image. Then we use size criteria to find out character candidates from all connected areas and use collinear criteria to group separate character candidates into text strings. Finally we analyze text strings according to text patterns summarized from dimension texts and give out the recognition result.