One of the main drivers of the recent advances in authorship verification is the PAN large-scale authorship dataset.Despite generating significant progress in the field, inconsistent performance differences between the closed and open test sets have been reported.To this end, we improve the experimental setup by proposing five new public splits over the PAN dataset, specifically designed to isolate and identify biases related to the text topic and to the author's writing style.We evaluate several BERT-like baselines on these splits, showing that such models are competitive with authorship verification state-of-the-art methods.Furthermore, using explainable AI, we find that these baselines are biased towards named entities.We show that models trained without the named entities obtain better results and generalize better when tested on DarkReddit, our new dataset for authorship verification.Test split O2D2 * O2D2 BERT Naive † Comp.† Closed 93.5 96.4 95.6 75.6 72.2 Clopen 94.0 96.0 97.4 74.1 71.1 Open UA 92.6 92.6 90.2 78.6 68.5 Open UF 91.4 95.1 91.6 79.9 79.0 Open All 80.6 67.5 88.7 75.6 76.9 PAN Closed 93.3 93.5 -74.7 74.2 PAN Open 93.3 94.4 -75.3 74.

Analyzing the distribution shift of data is a growing research direction in nowadays Machine Learning (ML), leading to emerging new benchmarks that focus on providing a suitable scenario for studying the generalization properties of ML models. The existing benchmarks are focused on supervised learning, and to the best of our knowledge, there is none for unsupervised learning. Therefore, we introduce an unsupervised anomaly detection benchmark with data that shifts over time, built over Kyoto-2006+, a traffic dataset for network intrusion detection. This type of data meets the premise of shifting the input distribution: it covers a large time span (10 years), with naturally occurring changes over time (e.g. users modifying their behavior patterns, and software updates). We first highlight the non-stationary nature of the data, using a basic per-feature analysis, t-SNE, and an Optimal Transport approach for measuring the overall distribution distances between years. Next, we propose AnoShift, a protocol splitting the data in IID, NEAR, and FAR testing splits. We validate the performance degradation over time with diverse models, ranging from classical approaches to deep learning. Finally, we show that by acknowledging the distribution shift problem and properly addressing it, the performance can be improved compared to the classical training which assumes independent and identically distributed data (on average, by up to 3% for our approach). Dataset and code are available at https://github.com/bit-ml/AnoShift/.

The Dark Web represents a hotbed for illicit activity, where users communicate on different market forums in order to exchange goods and services. Law enforcement agencies benefit from forensic tools that perform authorship analysis, in order to identify and profile users based on their textual content. However, authorship analysis has been traditionally studied using corpora featuring literary texts such as fragments from novels or fan fiction, which may not be suitable in a cybercrime context. Moreover, the few works that employ authorship analysis tools for cybercrime prevention usually employ ad-hoc experimental setups and datasets. To address these issues, we release VeriDark: a benchmark comprised of three large scale authorship verification datasets and one authorship identification dataset obtained from user activity from either Dark Web related Reddit communities or popular illicit Dark Web market forums. We evaluate competitive NLP baselines on the three datasets and perform an analysis of the predictions to better understand the limitations of such approaches. We make the datasets and baselines publicly available at https://github.com/bit-ml/VeriDark .

Leveraging deep learning models for Anomaly Detection (AD) has seen widespread use in recent years due to superior performances over traditional methods.Recent deep methods for anomalies in images learn better features of normality in an end-to-end self-supervised setting.These methods train a model to discriminate between different transformations applied to visual data and then use the output to compute an anomaly score.We use this approach for AD in text, by introducing a novel pretext task on text sequences.We learn our DATE model end-to-end, enforcing two independent and complementary self-supervision signals, one at the token-level and one at the sequencelevel.Under this new task formulation, we show strong quantitative and qualitative results on the 20Newsgroups and AG News datasets.In the semi-supervised setting, we outperform state-of-the-art results by +13.5% and +6.9%, respectively (AUROC).In the unsupervised configuration, DATE surpasses all other methods even when 10% of its training data is contaminated with outliers (compared with 0% for the others).

Radu Cristian Alexandru Iacob, Florin Brad, Elena-Simona Apostol, Ciprian-Octavian Truică, Ionel Alexandru Hosu, Traian Rebedea. Proceedings of the 28th International Conference on Computational Linguistics. 2020.

We propose a sketch-based two-step neural model for generating structured queries (SQL) based on a user’s request in natural language. The sketch is obtained by using placeholders for specific entities in the SQL query, such as column names, table names, aliases and variables, in a process similar to semantic parsing. The first step is to apply a sequence-to-sequence (SEQ2SEQ) model to determine the most probable SQL sketch based on the request in natural language. Then, a second network designed as a dual-encoder SEQ2SEQ model using both the text query and the previously obtained sketch is employed to generate the final SQL query. Our approach shows improvements over previous approaches on two recent large datasets (WikiSQL and SENLIDB) suitable for data-driven solutions for natural language interfaces for databases.

Recent advances in neural code generation have incorporated syntax to improve the generation of the target code based on the user's request in natural language. We adapt the model of [1] to the Natural Language Interface to Databases (NLIDB) problem by taking into account the database schema. We evaluate our model on the recently introduced WIKISQL and SENLIDB datasets. Our results show that the syntax-guided model outperforms a simple sequence-to-sequence (SEQ2SEQ) baseline on WIKISQL, but has trouble with the SENLIDB dataset due to its complexity.

Progress in natural language interfaces to databases (NLIDB) has been slow mainly due to linguistic issues (such as language ambiguity) and domain portability. Moreover, the lack of a large corpus to be used as a standard benchmark has made data-driven approaches difficult to develop and compare. In this paper, we revisit the problem of NLIDBs and recast it as a sequence translation problem. To this end, we introduce a large dataset extracted from the Stack Exchange Data Explorer website, which can be used for training neural natural language interfaces for databases. We also report encouraging baseline results on a smaller manually annotated test corpus, obtained using an attention-based sequence-to-sequence neural network.

Progress in statistical paraphrase generation has been hindered for a long time by the lack of large monolingual parallel corpora.In this paper, we adapt the neural machine translation approach to paraphrase generation and perform transfer learning from the closely related task of entailment generation.We evaluate the model on the Microsoft Research Paraphrase (MSRP) corpus and show that the model is able to generate sentences that capture part of the original meaning, but fails to pick up on important words or to show large lexical variation.