Masked language models (MLMs) have contributed to drastic performance improvements with regard to zero anaphora resolution (ZAR).To further improve this approach, in this study, we made two proposals.The first is a new pretraining task that trains MLMs on anaphoric relations with explicit supervision, and the second proposal is a new finetuning method that remedies a notorious issue, the pretrainfinetune discrepancy.Our experiments on Japanese ZAR demonstrated that our two proposals boost the state-of-the-art performance, and our detailed analysis provides new insights on the remaining challenges.

Interpretable rationales for model predictions play a critical role in practical applications.In this study, we develop models possessing interpretable inference process for structured prediction.Specifically, we present a method of instance-based learning that learns similarities between spans.At inference time, each span is assigned a class label based on its similar spans in the training set, where it is easy to understand how much each training instance contributes to the predictions.Through empirical analysis on named entity recognition, we demonstrate that our method enables to build models that have high interpretability without sacrificing performance.

One critical issue of zero anaphora resolution (ZAR) is the scarcity of labeled data.This study explores how effectively this problem can be alleviated by data augmentation.We adopt a state-ofthe-art data augmentation method, called the contextual data augmentation (CDA), that generates labeled training instances using a pretrained language model.The CDA has been reported to work well for several other natural language processing tasks, including text classification and machine translation (Kobayashi, 2018;Wu et al., 2019;Gao et al., 2019).This study addresses two underexplored issues on CDA, that is, how to reduce the computational cost of data augmentation and how to ensure the quality of the generated data.We also propose two methods to adapt CDA to ZAR: [MASK]-based augmentation and linguistically-controlled masking.Consequently, the experimental results on Japanese ZAR show that our methods contribute to both the accuracy gain and the computation cost reduction.Our closer analysis reveals that the proposed method can improve the quality of the augmented training data when compared to the conventional CDA.