The attention mechanism is a powerful and effective method utilized in natural language processing.However, it has been observed that this method is insensitive to positional information.Although several studies have attempted to improve positional encoding and investigate the influence of word order perturbation, it remains unclear how positional encoding impacts NLP models from the perspective of word order.In this paper, we aim to shed light on this problem by analyzing the working mechanism of the attention module and investigating the root cause of its inability to encode positional information.Our hypothesis is that the insensitivity can be attributed to the weight sum operation utilized in the attention module.To verify this hypothesis, we propose a novel weight concatenation operation and evaluate its efficacy in neural machine translation tasks.Our enhanced experimental results not only reveal that the proposed operation can effectively encode positional information but also confirm our hypothesis.

With the rapid development of data stream, multi-label algorithms for mining dynamic data become more and more important. At the same time, when data distribution changes, concept drift will occur, which will make the existing classification models lose effectiveness. Ensemble methods have been used for multi-label classification, but few methods consider both the accuracy and diversity of base classifiers. To address the above-mentioned problem, a Weighted Ensemble classification algorithm based on Nearest Neighbors for Multi-Label data stream (WENNML) is proposed. WENNML uses data blocks to train Active candidate Ensemble Classifiers (AEC) and Passive candidate Ensemble Classifiers (PEC). The base classifiers of AEC and PEC are dynamically updated using geometric and diversity weighting methods. When the difference value between the number of current instances and the number of warning instances reaches the passive warning value, the algorithm selects the optimal base classifiers from AEC and PEC according to the subset accuracy and hamming score and puts them into the predictive ensemble classifiers. Experiments are carried out on 12 kinds of datasets with 9 comparison algorithms. The results show that WENNML achieves the best average rankings among the four evaluation metrics.

Abstract At present, concept drift in the nonstationary data stream is showing trends with different speeds and different degrees of severity, which has brought great challenges to many fields like data mining and machine learning. In the past two decades, a lot of methods dedicated to handling concept drift in the nonstationary data stream have emerged. A novel perspective is proposed to classify these methods, and the current concept drift handling methods are comprehensively explained from the active handling methods and the passive handling methods. In particular, active handling methods are analyzed from the perspective of handling one specific type of concept drift and handling multiple types of concept drift, and passive handling methods are analyzed from the perspective of single learner and ensemble learning. Many concept drift handling methods in this survey are analyzed and summarized in terms of the comparing algorithms, learning model, applicable drift type, advantages, and disadvantages of the algorithms. Finally, further research directions are given, including the active and passive mixing methods, class imbalance, the existence of novel class in the data stream, and the noise in the data stream.