Multimodal relation extraction is a fundamental task of multimodal information extraction. Recent studies have shown promising results by integrating hierarchical visual features from local regions, like image patches, to the broader global regions that form the entire image. However, research to date has largely ignored the understanding of how hierarchical visual semantics are represented and the characteristics that can benefit relation extraction. To bridge this gap, we propose a novel two-stage hierarchical visual context fusion transformer incorporating the mixture of multimodal experts framework to effectively represent and integrate hierarchical visual features into textual semantic representations. In addition, we introduce the concept of hierarchical tracking maps to facilitate the understanding of the intrinsic mechanisms of image information processing involved in multimodal models. We thoroughly investigate the implications of hierarchical visual contexts through four dimensions: performance evaluation, the nature of auxiliary visual information, the patterns observed in the image encoding hierarchy, and the significance of various visual encoding levels. Empirical studies show that our approach achieves new state-of-the-art performance on the MNRE dataset.

Opinion target extraction (OTE) or aspect extraction (AE) is a fundamental task in opinion mining that aims to extract the targets (or aspects) on which opinions have been expressed. Recent work focus on cross-domain OTE, which is typically encountered in real-world scenarios, where the testing and training distributions differ. Most methods use domain adversarial neural networks that aim to reduce the domain gap between the labelled source and unlabelled target domains to improve target domain performance. However, this approach only aligns feature distributions and does not account for class-wise feature alignment, leading to suboptimal results. Semi-supervised learning (SSL) has been explored as a solution, but is limited by the quality of pseudo-labels generated by the model. Inspired by the theoretical foundations in domain adaptation [2], we propose a new SSL approach that opts for selecting target samples whose model output from a domain-specific teacher and student network disagree on the unlabelled target data, in an effort to boost the target domain performance. Extensive experiments on benchmark cross-domain OTE datasets show that this approach is effective and performs consistently well in settings with large domain shifts.

Word Sense Disambiguation (WSD) which aims to identify the correct sense of a target word appearing in a specific context is essential for web text analysis. The use of glosses has been explored as a means for WSD. However, only a few works model the correlation between the target context and gloss. We add to the body of literature by presenting a model that employs a multi-head attention mechanism on deep contextual features of the target word and candidate glosses to refine the target word embedding. Furthermore, to encourage the model to learn the relevant part of target features that align with the correct gloss, we recursively alternate attention on target word features and that of candidate glosses to gradually extract the relevant contextual features of the target word, refining its representation and strengthening the final disambiguation results. Empirical studies on the five most commonly used benchmark datasets show that our proposed model is effective and achieves state-of-the-art results.

Information extraction aims at extracting entities, relations, and so on, in text to support information retrieval systems. To extract information, researchers have considered multitask learning (ML) approaches. The conventional ML approach learns shared features across tasks, with the assumption that these features capture sufficient task interactions to learn expressive shared representations for task classification. However, such an assumption is flawed in different perspectives. First, the shared representation may contain noise introduced by another task; tasks coupled for multitask learning may have different complexities but this approach treats all tasks equally; the conventional approach has a flat structure that hinders the learning of explicit interactions. This approach, however, learns implicit interactions across tasks and often has a generalization ability that has benefited the learning of multitasks. In this article, we take advantage of implicit interactions learned by conventional approaches while alleviating the issues mentioned above by developing a Recurrent Interaction Network with an effective Early Prediction Integration (RIN-EPI) for multitask learning. Specifically, RIN-EPI learns implicit and explicit interactions across two different but related tasks. To effectively learn explicit interactions across tasks, we consider the correlations among the outputs of related tasks. It is, however, obvious that task outputs are unobservable during training, so we leverage the predictions at intermediate layers (referred to as early predictions) as proxies as well as shared features across tasks to learn explicit interactions through attention mechanisms and sequence learning models. By recurrently learning explicit interactions, we gradually improve predictions for the individual tasks in the multitask learning. We demonstrate the effectiveness of RIN-EPI on the learning of two mainstream multitasks for information extraction: (1) entity recognition and relation classification and (2) aspect and opinion term co-extraction. Extensive experiments demonstrate the effectiveness of the RIN-EPI architecture, where we achieve state-of-the-art results on several benchmark datasets.

Supervised learning relies heavily on readily available labelled data to infer an effective classification function. However, proposed methods under the supervised learning paradigm are faced with the scarcity of labelled data within domains, and are not generalized enough to adapt to other tasks. Transfer learning has proved to be a worthy choice to address these issues, by allowing knowledge to be shared across domains and tasks. In this paper, we propose two transfer learning methods Anchored Model Transfer (AMT) and Soft Instance Transfer (SIT), which are both based on multi-task learning, and account for model transfer and instance transfer, and can be combined into a common framework. We demonstrate the effectiveness of AMT and SIT for aspect-level sentiment classification showing the competitive performance against baseline models on benchmark datasets. Interestingly, we show that the integration of both methods AMT+SIT achieves state-of-the-art performance on the same task.

Real-world knowledge bases such as DBPedia, Yago, and Freebase contain sparse linkage connectivity, which poses a severe challenge to link prediction between entities. To cope with such data scarcity issues, recent models have focused on learning interactions between entity pairs by means of relations that exist between them. However promising, some relations are associated with very few tail entities or head entities, resulting in poor estimation of the relation interaction between entities. In this article, we break the sole dependency of modeling relation interactions between entity pairs by associating a triple with pairwise embeddings, i.e., distributed vector representations for pairs of word-based entities and relation of a triple. We capture the interactions that exist between pairwise embeddings by means of a Pairwise Factorization Model that employs a factorization machine with relation attention. This approach allows parameters for related interactions to be estimated efficiently, ensuring that the pairwise embeddings are discriminative, providing strong supervisory signals for the decoding task of link prediction. The Pairwise Factorization Model we propose exploits a neural bag-of-words model as the encoder, which effectively encodes word-based entities into distributed vector representations for the decoder. The proposed model is simple and enjoys efficiency and capability, showing superior link prediction performance over state-of-the-art complex models on benchmark datasets DBPedia50K and FB15K-237.

Knowledge graphs such as DBPedia and Freebase contain sparse linkage connectivity, which poses severe challenge to link prediction between entities. In addressing this sparsity problem, our studies indicate that one needs to leverage model with low complexity to avoid overfitting the weak structural information in the graphs, requiring the simple models which can efficiently encode the entities and their description information and then effectively decode their relationships. In this paper, we present a simple and efficient model that can attain these two goals. Specifically, we use a bag-of-words model, where relevant words are aggregated using average pooling or a basic Graph Convolutional Network to encode entities into distributed embeddings. A factorization machine is then used to score the relationships between those embeddings to generate linkage predictions. Empirical studies on two real datasets confirms the efficiency of our proposed model and shows superior predictive performance over state-of-the-art approaches.