Knowledge graphs (KGs) can provide rich factual knowledge for language models , enhancing reasoning ability and interpretability . However, existing knowledge injection methods usually ignore the structured information in KGs. Using structured knowledge to enhance pre-trained language models (PLMs) still has a set of challenging issues, including resource consumption of knowledge retraining, heterogeneous information, and knowledge noise. To address these issues, we explore how to flexibly inject structured knowledge into frozen PLMs. Inspired by prompt learning, we propose a novel method K nowledge G raph Prompt (KG-Prompt), which for the first time encodes the KG as structured prompts to enhance the knowledge expression ability of PLMs. KG-Prompt consists of a compressed subgraph construction module and a KG prompt generation module. In the compressed subgraph construction module, we construct compressed subgraphs based on a path-weighting strategy to reduce knowledge noise. In the KG prompt generation module, we propose a multi-hop consistency optimization strategy to learn the representation of compressed subgraphs, and then generate KG prompts based on a knowledge mapper to solve the heterogeneous information problem. The KG prompts can be inserted into the input of PLMs expediently, which decouples from PLMs and the downstream model without knowledge retraining and reduces computational resources . Extensive experiments on three knowledge-driven natural language understanding tasks demonstrate that our approach effectively improves the knowledge reasoning ability of PLMs. Furthermore, we provide a detailed analysis of different KG prompts and discuss the interpretability and generalizability of the proposed method.

Generative query rewrite generates reconstructed query rewrites using the conversation history while rely heavily on gold rewrite pairs that are expensive to obtain. Recently, few-shot learning is gaining increasing popularity for this task, whereas these methods are sensitive to the inherent noise due to limited data size. Besides, both attempts face performance degradation when there exists language style shift between training and testing cases. To this end, we study low-resource generative conversational query rewrite that is robust to both noise and language style shift. The core idea is to utilize massive unlabeled data to make further improvements via a contrastive co-training paradigm. Specifically, we co-train two dual models (namely Rewriter and Simplifier) such that each of them provides extra guidance through pseudo-labeling for enhancing the other in an iterative manner. We also leverage contrastive learning with data augmentation, which enables our model pay more attention on the truly valuable information than the noise. Extensive experiments demonstrate the superiority of our model under both few-shot and zero-shot scenarios. We also verify the better generalization ability of our model when encountering language style shift.

The task of query rewrite aims to convert an in-context query to its fully-specified version where ellipsis and coreference are completed and referred-back according to the history context.Although much progress has been made, less efforts have been paid to real scenario conversations that involve drawing information from more than one modalities.In this paper, we propose the task of multimodal conversational query rewrite (McQR), which performs query rewrite under the multimodal visual conversation setting.We collect a largescale dataset named McQueen based on manual annotation, which contains 15k visual conversations and over 80k queries where each one is associated with a fully-specified rewrite version.In addition, for entities appearing in the rewrite, we provide the corresponding image box annotation.We then use the McQueen dataset to benchmark a state-of-the-art method for effectively tackling the McQR task, which is based on a multimodal pre-trained model with pointer generator.Extensive experiments are performed to demonstrate the effectiveness of our model on this task 1 .

Fine-grained classification is absorbed in recognizing the subordinate categories of one field, which need a large number of labeled images, while it is expensive to label these images. Utilizing web data has been an attractive option to meet the demands of training data for convolutional neural networks (CNNs), especially when the well-labeled data is not enough. However, directly training on such easily obtained images often leads to unsatisfactory performance due to factors such as noisy labels. This has been conventionally addressed by reducing the noise level of web data. In this paper, we take a fundamentally different view and propose an adversarial discriminative loss to advocate representation coherence between standard and web data. This is further encapsulated in a simple, scalable and end-to-end trainable multi-task learning framework. We experiment on three public datasets using large-scale web data to evaluate the effectiveness and generalizability of the proposed approach. Extensive experiments demonstrate that our approach performs favorably against the state-of-the-art methods.

Visual arts consist of various art forms, \eg painting, sculpture, architecture,~\etc, which not only enrich our lives but also involve works related to aesthetics, history, and culture. Different eras have different artistic appeals, and the art also has characteristics of each era in terms of expression and spiritual pursuit, in which style is an important attribute to describe visual arts. In order to recognize the style of visual arts more effectively, we present an end-to-end trainable architecture to learn a deep style representation. The main component of our architecture, adaptive cross-layer correlation, is inspired by the Gram matrix based correlation calculation. Our proposed method can adaptively weight features in different spatial locations based on their intrinsic similarity. Extensive experiments on three datasets demonstrate the superiority of the proposed method over several state-of-the-art methods.

Analyzing and categorizing the style of visual art images, especially paintings, is gaining popularity owing to its importance in understanding and appreciating the art. The evolution of painting style is both continuous, in a sense that new styles may inherit, develop or even mutate from their predecessors and multi-modal because of various issues such as the visual appearance, the birthplace, the origin time and the art movement. Motivated by this peculiarity, we introduce a novel knowledge distilling strategy to assist visual feature learning in the convolutional neural network for painting style classification. More specifically, a multi-factor distribution is employed as soft-labels to distill complementary information with visual input, which extracts from different historical context via label distribution learning. The proposed method is well-encapsulated in a multi-task learning framework which allows end-to-end training. We demonstrate the superiority of the proposed method over the state-of-the-art approaches on Painting91, OilPainting, and Pandora datasets.