Visual question generation (VQG) task aims to generate high-quality questions based on the input image. Current methods primarily focus on generating questions containing specified content utilizing answers or question types as constraints. However, these constraints make it challenging to control the topic of generated questions (e.g., conversation or test subject topics) for various applications. Thus, it is necessary to utilize topics as constraints to guide question generation. Considering that there are many topics and it is almost impossible for human annotations to cover them, we propose the cross-topic learning VQG (CTL-VQG) task, which aims to generate questions related to unseen topics in cross-topic scenarios. In this paper, we propose a knowledge-guided cross-topic visual question generation (KC-VQG) model to extract unseen topic-related information for question generation. Specifically, an image-topic feature extractor is introduced in our model to extract topic-related intuitive visual features; an image-topic knowledge extractor is used to extract and select the most appropriate topic-related implicit knowledge from large language models for generating questions. Extensive experiments show that our model outperforms baselines and can effectively generate unseen topic-related questions in cross-topic scenarios.

Cross-domain aspect-based sentiment analysis has recently attracted significant attention, which can effectively alleviate the problem of lacking large-scale labeled data for supervised learning based methods. Most of current methods mainly focus on extracting domain-shared syntactic features to conduct the domain adaptation. Due to the language and syntax are diverse between domains, these methods lack generalization and even lead to syntactic transfer errors. External knowledge graphs have rich domain commonsense and share the relational structures between source and target domains. The domain-shared relational structure can effectively bridge the gap across domains and solve the problem of syntactic transfer errors. Moreover, not all the introduced external knowledge is equally important for the cross-domain aspect-based sentiment analysis. Motivated by these, we propose a knowledge-enhanced and topic-guided cross domain aspect-based sentiment analysis model with the domain-shared commonsense relational structure learning module and the topic-guided knowledge attention module. Extensive experiments are conducted and the experimental results evaluate the effectiveness of our proposed model.

In natural language processing, relation extraction (RE) is to detect and classify the semantic relationship of two given entities within a sentence. Previous RE methods consider only the textual contents and suffer performance decline in social media when texts lack contexts. Incorporating text-related visual information can supplement the missing semantics for relation extraction in social media posts. However, textual relations are usually abstract and of high-level semantics, which causes the semantic gap between visual contents and textual expressions. In this paper, we propose RECK - a neural network for relation extraction with cross-modal knowledge representations. Different from previous multimodal methods training a common subspace for all modalities, we bridge the semantic gaps by explicitly selecting knowledge paths from external knowledge through the cross-modal object-entity pairs. We further extend the paths into a knowledge graph, and adopt a graph attention network to capture the multi-grained relevant concepts which can provide higher level and key semantics information from external knowledge. Besides, we employ a cross-modal attention mechanism to align and fuse the multimodal information. Experimental results on a multimodal RE dataset show that our model achieves new state-of-the-art performance with knowledge evidence.

Infographics, which usually contain many well-designed visual elements, have significant advantages in delivering information efficiently and accurately. Previous research shows that proportion-related infographics make up the majority of all infographics. However, the creation of proportion-related infographics is difficult for general users. Recently, many researchers focus on generating infographics from the text with a single proportional fact. Our further research found that users tend to create infographics with multiple proportional facts. Existing research lacks modeling of relations between different facts, resulting in poor performance when generating infographics with multiple facts. In this paper, we model the relationship of different proportional facts based on the results of our investigation and design a deep learning-based model to classify them. At the same time, we also optimize the ability to extract multiple proportional facts from text. The experiments show that our model outperforms existing models when visualizing text with multiple proportional facts.

Aspect Based Sentiment Analysis (ABSA) aims to extract aspect terms and identify the sentiment polarities towards each extracted aspect term. Currently, syntactic information is seen as the bridge for the domain adaptation and achieves remarkable performance. However, the transferable syntactic knowledge is complex and diverse, which causes the transfer error problem in domain adaptation. In our paper, we propose a domain-shared relational structure incorporated cross-domain ABSA model. The experimental results show the effectiveness of our model.

Relation Classification (RC) is an important task in information extraction. In most real-world scenarios, the frequency of relations often follows a long-tailed and open-ended distribution. However, current efforts mainly focus on the partial frequency distribution of relations, which is limited in real-world applications. Meanwhile, prototypical network achieves remarkable performance among fields of deep supervised learning, few-shot learning and open set learning. Nevertheless, in the open world environment, it still suffers from the incompatible feature embedding problem as the novel and unknown relations come in. To address these problems, we propose an Open Generalized Prototypical Network with task-adaptive feature fusion for the open generalized few-shot relation classification. Extensive experiments are conducted on public large-scale datasets and our proposed model obtains the better performances.

Traditional sequence tagging methods for named entity recognition (NER) face challenges when handling nested entities, where an entity is nested in another. Most previous methods for nested NER ignore the effect of entity boundary information or type information. Considering that entity boundary information and type information can be utilized to improve the performance of boundary detection, we propose a nested NER model with a multi-agent communication module. The type tagger and boundary tagger in the multi-agent communication module iteratively utilize the information from each other, which improves the boundary detection and the final performance of nested NER. Empirical experiments conducted on two nested NER datasets show the effectiveness of our model.

The interaction between named entity recognition and relation classification is quite essential for the extraction of relational triplets. However, most of jointly extraction works only consider unidirectional interaction between the two sub-tasks. They even neglect the interactive information totally. In order to tackle these problems, we propose a novel unified joint extraction model which considers bidirection-interactive information between the two subtasks. Our model consists of two modules. The first module utilizes Bi-LSTM and GCN to capture the sequential and the structure-semantic features of a sentence, The second module utilizes two layers to capture bidirection-interactive information between the two subtasks and generates relational triplets respectively. The experimental results show that our proposed model outperforms the state-of-the-art models on two public datasets.

Named entity recognition (NER) is a basic task in natural language processing. Traditionally, sequence labeling methods are applied to named entity recognition and achieve good performance. However, sequence labeling methods can not be straightly applied to recognize nested named entities where an entity is included in another entity. Recently, some new methods are proposed for nested named entity recognition. Most of them ignore that entity type information can help recognize entity boundaries or ignore that entity boundary information can help recognize entity type, which limits the performance of nested NER. Considering the effect of entity type information and entity boundary information, in this paper, we propose a multi-agent communication module to utilize these two kinds of information. Our multi-agent communication module contains a type labeling agent and a boundary labeling agent. The type labeling agent can utilize boundary information from boundary labeling agent to recognize entity type. And the boundary labeling agent can utilize type information from type labeling agent to recognize entity boundaries. They communicate and collaborate iteratively to finish the entity boundary recognition. Compared with previous methods, with the assist of entity type information and entity boundary information, the performance of boundary recognition improves. The improvement of boundary recognition is beneficial to recognizing nested named entities, which improves the performance of nested named entity recognition. Empirical experiments are conducted on three nested NER datasets. And the experimental results show the effectiveness of our model.

Relation Classification (RC) plays an important role in natural language processing (NLP).Current conventional supervised and distantly supervised RC models always make a closed-world assumption which ignores the emergence of novel relations in an open environment.To incrementally recognize the novel relations, current two solutions (i.e, re-training and lifelong learning) are designed but suffer from the lack of large-scale labeled data for novel relations.Meanwhile, prototypical network enjoys better performance on both fields of deep supervised learning and few-shot learning.However, it still suffers from the incompatible feature embedding problem when the novel relations come in.Motivated by them, we propose a two-phase prototypical network with prototype attention alignment and triplet loss to dynamically recognize the novel relations with a few support instances meanwhile without catastrophic forgetting.Extensive experiments are conducted to evaluate the effectiveness of our proposed model.