The advent of scalable deep models and large datasets has improved the performance of Neural Machine Translation (NMT). Knowledge Distillation (KD) enhances efficiency by transferring knowledge from a teacher model to a more compact student model. However, KD approaches to Transformer architecture often rely on heuristics, particularly when deciding which teacher layers to distill from. In this paper, we introduce the “Align-to-Distill” (A2D) strategy, designed to address the feature mapping problem by adaptively aligning student attention heads with their teacher counterparts during training. The Attention Alignment Module (AAM) in A2D performs a dense head-by-head comparison between student and teacher attention heads across layers, turning the combinatorial mapping heuristics into a learning problem. Our experiments show the efficacy of A2D, demonstrating gains of up to +3.61 and +0.63 BLEU points for WMT-2022 De→Dsb and WMT-2014 En→De, respectively, compared to Transformer baselines.The code and data are available at https://github.com/ncsoft/Align-to-Distill.

Donghyun Kim, Youbin Ahn, Wongyu Kim, Chanhee Lee, Kyungchan Lee, Kyong-Ho Lee, Jeonguk Kim, Donghoon Shin, Yeonsoo Lee. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.

Donghyun Kim, Youbin Ahn, Chanhee Lee, Wongyu Kim, Kyong-Ho Lee, Donghoon Shin, Yeonsoo Lee. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.

Humans usually have conversations by making use of prior knowledge about a topic and background information of the people whom they are talking to. However, existing conversational agents and datasets do not consider such comprehensive information, and thus they have a limitation in generating the utterances where the knowledge and persona are fused properly. To address this issue, we introduce a call For Customized conversation (FoCus) dataset where the customized answers are built with the user's persona and Wikipedia knowledge. To evaluate the abilities to make informative and customized utterances of pre-trained language models, we utilize BART and GPT-2 as well as transformer-based models. We assess their generation abilities with automatic scores and conduct human evaluations for qualitative results. We examine whether the model reflects adequate persona and knowledge with our proposed two sub-tasks, persona grounding (PG) and knowledge grounding (KG). Moreover, we show that the utterances of our data are constructed with the proper knowledge and persona through grounding quality assessment.

Both entity typing and relation extraction from text corpora are widely used to identify the semantic types of an entity and a relation in a knowledge graph (KG). Most existing approaches rely on a pre-defined set of entity types and relation types in a KG. They thus cannot map entity mentions (relation mentions) to unseen entity types (relation types). To fundamentally overcome the limitations, we should add new semantic types of entities and relations to a KG schema. However, schema expansion traditionally requires manual conceptualization through a user’s observation on the text corpus while assuming the existence of suitable target KG schemas. In this work, we propose anActive learning framework forKnowledge graphSchemaExpansion (AKSE), which can generate a new semantic type for KG schemas, without depending on a set of target schemas and human users’ observation. Specifically, a granularity based active learning algorithm determines whether a KG schema requires new semantic types or not. We also introduce a KG schema attention-based neural method which assigns semantic types to the entities and relationships extracted. To the best of our knowledge, our work is the first study to expand a KG schema with active learning.

This paper presents an efficient method of extracting n-ary relations from multiple sentences which is called Entity-path and Discourse relation-centric Relation Extractor (EDCRE). Unlike previous approaches, the proposed method focuses on an entity link, which consists of dependency edges between entities, and discourse relations between sentences. Specifically, the proposed model consists of two main sub-models. The first one encodes sentences with a higher weight on the entity link while considering the other edges with an attention mechanism. To consider various latent discourse relations between sentences, the second sub-model encodes discourse relations between adjacent sentences considering the contents of each sentence. Experiment results on the cross-sentence relation extraction dataset, PubMed, and the document-level relation extraction dataset, DocRED, show that the proposed model outperforms state-of-the-art methods of extracting relations across sentences. Furthermore, ablation study proves that both the two main sub-models have noticeable effect on the relation extraction task.