Self-supervised objectives have driven major advances in NLP by leveraging large-scale unlabeled data, but such resources are scarce for many of the world's languages.Surprisingly, they have not been explored much for characterlevel tasks, where smaller amounts of data have the potential to be beneficial.We investigate the effectiveness of self-supervised auxiliary tasks for morphological inflection -a character-level task highly relevant for language documentation -in extremely low-resource settings, training encoder-decoder transformers for 19 languages and 13 auxiliary objectives.Autoencoding yields the best performance when unlabeled data is very limited, while character masked language modeling (CMLM) becomes more effective as data availability increases.Though objectives with stronger inductive biases influence model predictions intuitively, they rarely outperform standard CMLM.However, sampling masks based on known morpheme boundaries consistently improves performance, highlighting a promising direction for low-resource morphological modeling.

We present NN-RANK, an algorithm for ranking source languages for cross-lingual transfer, which leverages hidden representations from multilingual models and unlabeled targetlanguage data.We experiment with two pretrained multilingual models and two tasks: partof-speech tagging (POS) and named entity recognition (NER).We consider 51 source languages and evaluate on 56 and 72 target languages for POS and NER, respectively.When using in-domain data, NN-RANK beats stateof-the-art baselines that leverage lexical and linguistic features, with average improvements of up to 35.56 NDCG for POS and 18.14 NDCG for NER.As prior approaches can fall back to language-level features if target language data is not available, we show that NN-RANK remains competitive using only the Bible, an out-of-domain corpus available for a large number of languages.Ablations on the amount of unlabeled target data show that, for subsets consisting of as few as 25 examples, NN-RANK produces high-quality rankings which achieve 92.8% of the NDCG achieved using all available target data for ranking.POS test-all NER test-all

Hyperparameter tuning, the process of searching for suitable hyperparameters, becomes more difficult as the computing resources required to train neural networks continue to grow.This topic continues to receive little attention and discussion-much of it hearsaydespite its obvious importance.We attempt to formalize hyperparameter sensitivity using two metrics: similarity-based sensitivity and performance-based sensitivity.We then use these metrics to quantify two such claims: (1) transformers are more sensitive to hyperparameter choices than LSTMs and (2) transformers are particularly sensitive to batch size.We conduct experiments on two different characterlevel sequence-to-sequence tasks and find that, indeed, the transformer is slightly more sensitive to hyperparameters according to both of our metrics.However, we do not find that it is more sensitive to batch size in particular.

Pretrained transformers such as BERT (Devlin et al., 2019) have been shown to be effective in many natural language tasks.However, they are under-explored for character-level sequence-tosequence tasks.In this work, we investigate pretraining transformers for the character-level task of morphological inflection in several languages.We compare various training setups and secondary tasks where unsupervised data taken directly from the target task is used.We show that training on secondary unsupervised tasks increases inflection performance even without any external data, suggesting that models learn from additional unsupervised tasks themselves-not just from additional data.We also find that this does not hold true for specific combinations of secondary task and training setup, which has interesting implications for unsupervised training and denoising objectives in character-level tasks.

Neural networks have long been at the center of a debate around the cognitive mechanism by which humans process inflectional morphology.This debate has gravitated into NLP by way of the question: Are neural networks a feasible account for human behavior in morphological inflection?We address that question by measuring the correlation between human judgments and neural network probabilities for unknown word inflections.We test a larger range of architectures than previously studied on two important tasks for the cognitive processing debate: English past tense, and German number inflection.We find evidence that the Transformer may be a better account of human behavior than LSTMs on these datasets, and that LSTM features known to increase inflection accuracy do not always result in more human-like behavior.