Zihan Liu, Zewei Sun, Shanbo Cheng, Shujian Huang, Mingxuan Wang. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

In this paper, a discretization‐free approach based on the physics‐informed neural network (PINN) is proposed for solving the forward and inverse problems governed by the nonlinear convection‐diffusion‐reaction (CDR) systems. By embedding physical information described by the CDR system in the feedforward neural networks, PINN is trained to approximate the solution of the system without the need of labeled data. The good performance of PINN in solving the forward problem of the nonlinear CDR systems is verified by studying the problems of gas‐solid adsorption and autocatalytic reacting flow. For CDR systems with different Péclet number, PINN can largely eliminate the numerical diffusion and unphysical oscillations in traditional numerical methods caused by high Péclet number. Meanwhile, the PINN framework is implemented to solve the inverse problem of nonlinear CDR systems and the results show that the unknown parameters can be effectively recognized even with high noisy data. It is concluded that the established PINN algorithm has good accuracy, convergence, and robustness for both the forward and inverse problems of CDR systems.

Recent studies show that the attention heads in Transformer are not equal. We relate this phenomenon to the imbalance training of multi-head attention and the model dependence on specific heads. To tackle this problem, we propose a simple masking method: HeadMask, in two specific ways. Experiments show that translation improvements are achieved on multiple language pairs. Subsequent empirical analyses also support our assumption and confirm the effectiveness of the method.

Transformer model (Vaswani et al. 2017) has been widely used in machine translation tasks and obtained state-of-the-art results. In this paper, we report an interesting phenomenon in its encoder-decoder multi-head attention: different attention heads of the final decoder layer align to different word translation candidates. We empirically verify this discovery and propose a method to generate diverse translations by manipulating heads. Furthermore, we make use of these diverse translations with the back-translation technique for better data augmentation. Experiment results show that our method generates diverse translations without a severe drop in translation quality. Experiments also show that back-translation with these diverse translations could bring a significant improvement in performance on translation tasks. An auxiliary experiment of conversation response generation task proves the effect of diversity as well.