A wide variety of neural-network architectures have been proposed for the task of Chinese word segmentation.Surprisingly, we find that a bidirectional LSTM model, when combined with standard deep learning techniques and best practices, can achieve better accuracy on many of the popular datasets as compared to models based on more complex neuralnetwork architectures.Furthermore, our error analysis shows that out-of-vocabulary words remain challenging for neural-network models, and many of the remaining errors are unlikely to be fixed through architecture changes.Instead, more effort should be made on exploring resources for further improvement.

Current state-of-the-art semantic role labeling (SRL) uses a deep neural network with no explicit linguistic features.However, prior work has shown that gold syntax trees can dramatically improve SRL decoding, suggesting the possibility of increased accuracy from explicit modeling of syntax.In this work, we present linguistically-informed self-attention (LISA): a neural network model that combines multi-head self-attention with multi-task learning across dependency parsing, part-ofspeech tagging, predicate detection and SRL.Unlike previous models which require significant pre-processing to prepare linguistic features, LISA can incorporate syntax using merely raw tokens as input, encoding the sequence only once to simultaneously perform parsing, predicate detection and role labeling for all predicates.Syntax is incorporated by training one attention head to attend to syntactic parents for each token.Moreover, if a high-quality syntactic parse is already available, it can be beneficially injected at test time without re-training our SRL model.In experiments on CoNLL-2005 SRL, LISA achieves new state-of-the-art performance for a model using predicted predicates and standard word embeddings, attaining 2.5 F1 absolute higher than the previous state-of-the-art on newswire and more than 3.5 F1 on outof-domain data, nearly 10% reduction in error.On ConLL-2012 English SRL we also show an improvement of more than 2.5 F1.LISA also out-performs the state-of-the-art with contextually-encoded (ELMo) word representations, by nearly 1.0 F1 on news and more than 2.0 F1 on out-of-domain text.

We address fine-grained multilingual language identification: providing a language code for every token in a sentence, including codemixed text containing multiple languages.Such text is prevalent online, in documents, social media, and message boards.We show that a feed-forward network with a simple globally constrained decoder can accurately and rapidly label both codemixed and monolingual text in 100 languages and 100 language pairs.This model outperforms previously published multilingual approaches in terms of both accuracy and speed, yielding an 800x speed-up and a 19.5% averaged absolute gain on three codemixed datasets.It furthermore outperforms several benchmark systems on monolingual language identification.

We show that small and shallow feedforward neural networks can achieve near state-of-the-art results on a range of unstructured and structured language processing tasks while being considerably cheaper in memory and computational requirements than deep recurrent models.Motivated by resource-constrained environments like mobile phones, we showcase simple techniques for obtaining such small neural network models, and investigate different tradeoffs when deciding how to allocate a small memory budget.

Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, Michael Collins. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2016.

Traditional syntax models typically leverage part-of-speech (POS) information by constructing features from hand-tuned templates.We demonstrate that a better approach is to utilize POS tags as a regularizer of learned representations.We propose a simple method for learning a stacked pipeline of models which we call "stack-propagation".We apply this to dependency parsing and tagging, where we use the hidden layer of the tagger network as a representation of the input tokens for the parser.At test time, our parser does not require predicted POS tags.On 19 languages from the Universal Dependencies, our method is 1.3% (absolute) more accurate than a state-of-the-art graph-based approach and 2.7% more accurate than the most comparable greedy model.* Research conducted at Google.

David Weiss, Chris Alberti, Michael Collins, Slav Petrov. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015.

We extend and improve upon recent work in structured training for neural network transition-based dependency parsing.We do this by experimenting with novel features, additional transition systems and by testing on a wider array of languages.In particular, we introduce set-valued features to encode the predicted morphological properties and part-ofspeech confusion sets of the words being parsed.We also investigate the use of joint parsing and partof-speech tagging in the neural paradigm.Finally, we conduct a multi-lingual evaluation that demonstrates the robustness of the overall structured neural approach, as well as the benefits of the extensions proposed in this work.Our research further demonstrates the breadth of the applicability of neural network methods to dependency parsing, as well as the ease with which new features can be added to neural parsing models.