This work presents a sparse-attention Transformer architecture for modeling documents that contain large tables.Tables are ubiquitous on the web, and are rich in information.However, more than 20% of relational tables on the web have 20 or more rows (Cafarella et al., 2008), and these large tables present a challenge for current Transformer models, which are typically limited to 512 tokens.Here we propose MATE, a novel Transformer architecture designed to model the structure of web tables.MATE uses sparse attention in a way that allows heads to efficiently attend to either rows or columns in a table.This architecture scales linearly with respect to speed and memory, and can handle documents containing more than 8000 tokens with current accelerators.MATE also has a more appropriate inductive bias for tabular data, and sets a new state-of-the-art for three table reasoning datasets.For HY-BRIDQA (Chen et al., 2020b), a dataset that involves large documents containing tables, we improve the best prior result by 19 points.

Jonathan Herzig, Thomas Müller, Syrine Krichene, Julian Eisenschlos. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Answering natural language questions over tables is usually seen as a semantic parsing task.To alleviate the collection cost of full logical forms, one popular approach focuses on weak supervision consisting of denotations instead of logical forms.However, training semantic parsers from weak supervision poses difficulties, and in addition, the generated logical forms are only used as an intermediate step prior to retrieving the denotation.In this paper, we present TAPAS, an approach to question answering over tables without generating logical forms.TAPAS trains from weak supervision, and predicts the denotation by selecting table cells and optionally applying a corresponding aggregation operator to such selection.TAPAS extends BERT's architecture to encode tables as input, initializes from an effective joint pre-training of text segments and tables crawled from Wikipedia, and is trained end-to-end.We experiment with three different semantic parsing datasets, and find that TAPAS outperforms or rivals semantic parsing models by improving state-of-the-art accuracy on SQA from 55.1 to 67.2 and performing on par with the state-of-the-art on WIKISQL and WIKITQ, but with a simpler model architecture.We additionally find that transfer learning, which is trivial in our setting, from WIK-ISQL to WIKITQ, yields 48.7 accuracy, 4.2 points above the state-of-the-art.

Thomas Mueller, Francesco Piccinno, Peter Shaw, Massimo Nicosia, Yasemin Altun. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

We present labeled morphological segmentation-an alternative view of morphological processing that unifies several tasks.We introduce a new hierarchy of morphotactic tagsets and CHIPMUNK, a discriminative morphological segmentation system that, contrary to previous work, explicitly models morphotactics.We show improved performance on three tasks for all six languages: (i) morphological segmentation, (ii) stemming and (iii) morphological tag classification.For morphological segmentation our method shows absolute improvements of 2-6 points F 1 over a strong baseline.

We present LEMMING, a modular loglinear model that jointly models lemmatization and tagging and supports the integration of arbitrary global features.It is trainable on corpora annotated with gold standard tags and lemmata and does not rely on morphological dictionaries or analyzers.LEMMING sets the new state of the art in token-based statistical lemmatization on six languages; e.g., for Czech lemmatization, we reduce the error by 60%, from 4.05 to 1.58.We also give empirical evidence that jointly modeling morphological tags and lemmata is mutually beneficial.

We present a comparative investigation of word representations for part-of-speech (POS) and morphological tagging, focusing on scenarios with considerable differences between training and test data where a robust approach is necessary.Instead of adapting the model towards a specific domain we aim to build a robust model across domains.We developed a test suite for robust tagging consisting of six languages and different domains.We find that representations similar to Brown clusters perform best for POS tagging and that word representations based on linguistic morphological analyzers perform best for morphological tagging.

In this paper we propose a method to increase dependency parser performance without using additional labeled or unlabeled data by refining the layer of predicted part-of-speech (POS) tags. We perform experiments on English and German and show significant improvements for both languages. The refinement is based on generative split-merge training for Hidden Markov models (HMMs).

Training higher-order conditional random fields is prohibitive for huge tag sets.We present an approximated conditional random field using coarse-to-fine decoding and early updating.We show that our implementation yields fast and accurate morphological taggers across six languages with different morphological properties and that across languages higher-order models give significant improvements over 1 st -order models.