Attention mechanism is an increasingly important approach in the field of natural language processing (NLP). In the attention-based named entity recognition (NER) model, most attention mechanisms can calculate attention coefficient to express the importance of sentence semantic information but cannot adjust the position distribution of contextual feature vectors in the semantic space. To address this issue, a radial basis function attention (RBF-attention) layer is proposed to adaptively regulate the position distribution of sequence contextual feature vectors, which can minimize the relative distance of within-category named entities and maximize the relative distance of between-category named entities in the semantic space. The experimental results on CoNLL2003 English and MSRA Chinese NER datasets indicate that the proposed model performs better than other baseline approaches without relying on any external feature engineering.

Due to the dynamic changes of network data over time, static intrusion detection systems cannot adapt well to the behavioral characteristics of the input network data, resulting in reduced detection accuracy. In addition, continuous input data streams will bring huge challenges to resource storage and computing costs. Therefore, we propose an intrusion detection method of dynamic ensemble incremental learning (DEIL-RVM), and realize a dynamically adjusted ensemble intrusion detection model. In which a new overall misclassification probability weight value (OMPW) based on incremental set or data chunk is designed as the basis for updating the ensemble model, and it can be used to prune and replace the poor base component in the ensemble model. We presented a probabilistic decision function taking into account the posterior probability of each base RVM model dividing the sample into each category. The RVM with high sparsity is used as the base component to obtain the good balance between the accuracy, robustness and resource consumption, which can sacrifice less time and storage cost in ensemble incremental learning while achieving higher detection accuracy and stability in network data streams.

In this paper, we propose BANG, a new pretraining model to Bridge the gap between Autoregressive (AR) and Non-autoregressive (NAR) Generation. AR and NAR generation can be uniformly regarded as to what extent previous tokens can be attended, and BANG bridges AR and NAR generation through designing a novel model structure for large-scale pre-training. A pretrained BANG model can simultaneously support AR, NAR, and semi-NAR generation to meet different requirements. Experiments on question generation (SQuAD 1.1), summarization (XSum), and dialogue generation (PersonaChat) show that BANG improves NAR and semi-NAR performance significantly as well as attaining comparable performance with strong AR pretrained models. Compared with the semi-NAR strong baselines, BANG achieves absolute improvements of 14.01 and 5.24 in the overall scores of SQuAD 1.1 and XSum, respectively. In addition, BANG achieves absolute improvements of 10.73, 6.39, and 5.90 in the overall scores of SQuAD, XSUM, and PersonaChat compared with the NAR strong baselines, respectively. Our code will be made publicly available.

Transformer model with multi-head attention requires caching intermediate results for efficient inference in generation tasks. However, cache brings new memory-related costs and prevents leveraging larger batch size for faster speed. We propose memory-efficient lossless attention (called EL-attention) to address this issue. It avoids heavy operations for building multi-head keys and values, cache for them is not needed. EL-attention constructs an ensemble of attention results by expanding query while keeping key and value shared. It produces the same result as multi-head attention with less GPU memory and faster inference speed. We conduct extensive experiments on Transformer, BART, and GPT-2 for summarization and question generation tasks. The results show EL-attention speeds up existing models by 1.6x to 5.3x without accuracy loss.

Reading long documents to answer opendomain questions remains challenging in natural language understanding.In this paper, we introduce a new model, called RikiNet, which reads Wikipedia pages for natural question answering.RikiNet contains a dynamic paragraph dual-attention reader and a multi-level cascaded answer predictor.The reader dynamically represents the document and question by utilizing a set of complementary attention mechanisms.The representations are then fed into the predictor to obtain the span of the short answer, the paragraph of the long answer, and the answer type in a cascaded manner.On the Natural Questions (NQ) dataset, a single RikiNet achieves 74.3 F1 and 57.9 F1 on longanswer and short-answer tasks.To our best knowledge, it is the first single model that outperforms the single human performance.Furthermore, an ensemble RikiNet obtains 76.1 F1 and 61.3 F1 on long-answer and shortanswer tasks, achieving the best performance on the official NQ leaderboard 1 .

In this paper, we propose a novel data augmentation method, referred to as Controllable Rewriting based Question Data Augmentation (CRQDA), for machine reading comprehension (MRC), question generation, and question-answering natural language inference tasks.We treat the question data augmentation task as a constrained question rewriting problem to generate context-relevant, high-quality, and diverse question data samples.CRQDA utilizes a Transformer autoencoder to map the original discrete question into a continuous embedding space.It then uses a pre-trained MRC model to revise the question representation iteratively with gradientbased optimization.Finally, the revised question representations are mapped back into the discrete space, which serve as additional question data.Comprehensive experiments on SQuAD 2.0, SQuAD 1.1 question generation, and QNLI tasks demonstrate the effectiveness of CRQDA 1 .

MOTIVATION: Flux estimation by using (13) C-labeling pattern information of metabolites is currently the only method that can give accurate, detailed quantification of all intracellular fluxes in the central metabolism of a microorganism. In essence, it corresponds to a constrained optimization problem which minimizes a weighted distance between measured and simulated results. Characteristics, such as existence of multiple local minima, non-linear and non-differentiable make this problem a special difficulty. RESULTS: In the present work, we propose an evolutionary-based global optimization algorithm taking advantage of the convex feature of the problem's solution space. Based on the characteristics of convex spaces, specialized initial population and evolutionary operators are designed to solve (13)C-based metabolic flux estimation problem robustly and efficiently. The algorithm was applied to estimate the central metabolic fluxes in Escherichia coli and compared with conventional optimization technique. Experimental results illustrated that our algorithm is capable of achieving fast convergence to good near-optima and maintaining the robust nature of evolutionary algorithms at the same time. AVAILABILITY: Available from the authors upon request.