Multi-party dialogues are more difficult for models to understand than one-to-one twoparty dialogues, since they involve multiple interlocutors, resulting in interweaving reply-to relations and information flows.To step over these obstacles, an effective way is to pre-train a model that understands the discourse structure of multi-party dialogues, namely, to whom each utterance is replying.However, due to the lack of explicitly annotated discourse labels in multi-party dialogue corpora, previous works fail to scale up the pre-training process by putting aside the unlabeled multi-party conversational data for nothing.To fully utilize the unlabeled data, we propose to treat the discourse structures as latent variables, then jointly infer them and pre-train the discourse-aware model by unsupervised latent variable inference methods.Experiments on multiple downstream tasks show that our pre-trained model outperforms strong baselines by large margins and achieves state-of-the-art (SOTA) results, justifying the effectiveness of our method.The official implementation of this paper is available at https://github.com/EricLee8/MPD_EMVI.

Dialogue response generation requires an agent to generate a response according to the current dialogue history, in terms of which twoparty dialogues have been well studied, but leaving a great gap for multi-party dialogues at the same time.Different from two-party dialogues where each response is a direct reply to its previous utterance, the addressee of a response utterance should be specified before it is generated in the multi-party scenario.Thanks to the huge amount of two-party conversational data, various pre-trained language models for two-party dialogue response generation have been proposed.However, due to the lack of annotated addressee labels in multi-party dialogue datasets, it is hard to use them to pre-train a response generation model for multi-party dialogues.To tackle this obstacle, we propose an Expectation-Maximization (EM) approach that iteratively performs the expectation steps to generate addressee labels, and the maximization steps to optimize a response generation model.Theoretical analyses and extensive experiments have justified the feasibility and effectiveness of our proposed method.The official implementation of this paper is available at https://github.com/EricLee8/MPDRG.

Using pre-trained language models to obtain code embeddings is a common and effective practice in the field of source code comprehension. However, language models pre-trained on natural language text fail to capture some intrinsic characteristics of code snippets since programming languages are in a quite different form from natural language. In this paper, we present Functionality-aware Code Embeddings (FaCE) in terms of contrastive learning. The key idea of this work is that when comprehending a code snippet, it is the functionality that counts rather than its semantic meaning that mainly comes from its entities (e.g. names of functions, variables and classes). We construct positive samples and hard negative samples according to the functionality of code snippets, then pre-train our model by standard contrastive learning framework. Experimental results and massive analysis on two code-related benchmarks have justified the effectiveness of our proposed FaCE by outperforming the baseline models with large margins.

Based on the tremendous success of pre-trained language models (PrLMs) for source code comprehension tasks, current literature studies either ways to further improve the performance (generalization) of PrLMs, or their robustness against adversarial attacks. However, they have to compromise on the trade-off between the two aspects and none of them consider improving both sides in an effective and practical way. To fill this gap, we propose Semantic-Preserving Adversarial Code Embeddings (SPACE) to find the worst-case semantic-preserving attacks while forcing the model to predict the correct labels under these worst cases. Experiments and analysis demonstrate that SPACE can stay robust against state-of-the-art attacks while boosting the performance of PrLMs for code.

Multi-turn dialogue modeling as a challenging branch of natural language understanding (NLU), aims to build representations for machines to understand human dialogues, which provides a solid foundation for multiple downstream tasks.Recent studies of dialogue modeling commonly employ pre-trained language models (PrLMs) to encode the dialogue history as successive tokens, which is insufficient in capturing the temporal characteristics of dialogues.Therefore, we propose Bidirectional Information Decoupling Network (BiDeN) as a universal dialogue encoder, which explicitly incorporates both the past and future contexts and can be generalized to a wide range of dialogue-related tasks.Experimental results on datasets of different downstream tasks demonstrate the universality and effectiveness of our BiDeN.The official implementation of BiDeN is available at https://github.com/ EricLee8/BiDeN.