Understanding tables is an important aspect of natural language understanding.Existing models for table understanding require linearization of the table structure, where row or column order is encoded as an unwanted bias.Such spurious biases make the model vulnerable to row and column order perturbations.Additionally, prior work has not thoroughly modeled the table structures or table-text alignments, hindering the table-text understanding ability.In this work, we propose a robust and structurally aware table-text encoding architecture TABLEFORMER, where tabular structural biases are incorporated completely through learnable attention biases.TABLEFORMER is (1) strictly invariant to row and column orders, and, (2) could understand tables better due to its tabular inductive biases.Our evaluations showed that TABLEFORMER outperforms strong baselines in all settings on SQA, WTQ and TABFACT table reasoning datasets, and achieves state-of-the-art performance on SQA, especially when facing answer-invariant row and column order perturbations (6% improvement over the best baseline), because previous SOTA models' performance drops by 4% -6% when facing such perturbations while TABLEFORMER is not affected.1

Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, Manaal Faruqui. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Dian Yu, Luheng He, Yuan Zhang, Xinya Du, Panupong Pasupat, Qi Li. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Natural language descriptions of user interface (UI) elements such as alternative text are crucial for accessibility and language-based interaction in general.Yet, these descriptions are constantly missing in mobile UIs.We propose widget captioning, a novel task for automatically generating language descriptions for UI elements from multimodal input including both the image and the structural representations of user interfaces.We collected a largescale dataset for widget captioning with crowdsourcing.Our dataset contains 162,859 language phrases created by human workers for annotating 61,285 UI elements across 21,750 unique UI screens.We thoroughly analyze the dataset, and train and evaluate a set of deep model configurations to investigate how each feature modality as well as the choice of learning strategies impact the quality of predicted captions.The task formulation and the dataset as well as our benchmark models contribute a solid basis for this novel multimodal captioning task that connects language and user interfaces.

Daniel Andor, Luheng He, Kenton Lee, Emily Pitler. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

We present a new large-scale corpus of Question-Answer driven Semantic Role Labeling (QA-SRL) annotations, and the first high-quality QA-SRL parser.Our corpus, QA-SRL Bank 2.0, consists of over 250,000 question-answer pairs for over 64,000 sentences across 3 domains and was gathered with a new crowd-sourcing scheme that we show has high precision and good recall at modest cost.We also present neural models for two QA-SRL subtasks: detecting argument spans for a predicate and generating questions to label the semantic relationship.The best models achieve question accuracy of 82.6% and span-level accuracy of 77.6% (under human evaluation) on the full pipelined QA-SRL prediction task.They can also, as we show, be used to gather additional annotations at low cost.

We introduce a multi-task setup of identifying and classifying entities, relations, and coreference clusters in scientific articles.We create SCIERC, a dataset that includes annotations for all three tasks and develop a unified framework called Scientific Information Extractor (SCIIE) for with shared span representations.The multi-task setup reduces cascading errors between tasks and leverages cross-sentence relations through coreference links.Experiments show that our multi-task model outperforms previous models in scientific information extraction without using any domain-specific features.We further show that the framework supports construction of a scientific knowledge graph, which we use to analyze information in scientific literature. 1 Extracting nodes (entities) The SCIIE model extracts entities, their relations, and coreference

We introduce a new deep learning model for semantic role labeling (SRL) that significantly improves the state of the art, along with detailed analyses to reveal its strengths and limitations.We use a deep highway BiLSTM architecture with constrained decoding, while observing a number of recent best practices for initialization and regularization.Our 8-layer ensemble model achieves 83.2 F1 on the CoNLL 2005 test set and 83.4 F1 on CoNLL 2012, roughly a 10% relative error reduction over the previous state of the art.Extensive empirical analysis of these gains show that (1) deep models excel at recovering long-distance dependencies but can still make surprisingly obvious errors, and (2) that there is still room for syntactic parsers to improve these results.

We introduce the first end-to-end coreference resolution model and show that it significantly outperforms all previous work without using a syntactic parser or handengineered mention detector.The key idea is to directly consider all spans in a document as potential mentions and learn distributions over possible antecedents for each.The model computes span embeddings that combine context-dependent boundary representations with a headfinding attention mechanism.It is trained to maximize the marginal likelihood of gold antecedent spans from coreference clusters and is factored to enable aggressive pruning of potential mentions.Experiments demonstrate state-of-the-art performance, with a gain of 1.5 F1 on the OntoNotes benchmark and by 3.1 F1 using a 5-model ensemble, despite the fact that this is the first approach to be successfully trained with no external resources.

This paper demonstrates that it is possible for a parser to improve its performance with a human in the loop, by posing simple questions to non-experts.For example, given the first sentence of this abstract, if the parser is uncertain about the subject of the verb "pose," it could generate the question What would pose something?with candidate answers this paper and a parser.Any fluent speaker can answer this question, and the correct answer resolves the original uncertainty.We apply the approach to a CCG parser, converting uncertain attachment decisions into natural language questions about the arguments of verbs.Experiments show that crowd workers can answer these questions quickly, accurately and cheaply.Our human-in-the-loop parser improves on the state of the art with less than 2 questions per sentence on average, with a gain of 1.7 F1 on the 10% of sentences whose parses are changed.

This paper introduces the task of questionanswer driven semantic role labeling (QA-SRL), where question-answer pairs are used to represent predicate-argument structure.For example, the verb "introduce" in the previous sentence would be labeled with the questions "What is introduced?", and "What introduces something?", each paired with the phrase from the sentence that gives the correct answer.Posing the problem this way allows the questions themselves to define the set of possible roles, without the need for predefined frame or thematic role ontologies.It also allows for scalable data collection by annotators with very little training and no linguistic expertise.We gather data in two domains, newswire text and Wikipedia articles, and introduce simple classifierbased models for predicting which questions to ask and what their answers should be.Our results show that non-expert annotators can produce high quality QA-SRL data, and also establish baseline performance levels for future work on this task.

Joint models of syntactic and semantic parsing have the potential to improve performance on both tasks-but to date, the best results have been achieved with pipelines.We introduce a joint model using CCG, which is motivated by the close link between CCG syntax and semantics.Semantic roles are recovered by labelling the deep dependency structures produced by the grammar.Furthermore, because CCG is lexicalized, we show it is possible to factor the parsing model over words and introduce a new A * parsing algorithmwhich we demonstrate is faster and more accurate than adaptive supertagging.Our joint model is the first to substantially improve both syntactic and semantic accuracy over a comparable pipeline, and also achieves state-of-the-art results for a nonensemble semantic role labelling model.

Most existing collaborative filtering models only consider the use of user feedback (e.g., ratings) and meta data (e.g., content, demographics). However, in most real world recommender systems, context information, such as time and social networks, are also very important factors that could be considered in order to produce more accurate recommendations. In this work, we address several challenges for the context aware movie recommendation tasks in CAMRa 2010: (1) how to combine multiple heterogeneous forms of user feedback? (2) how to cope with dynamic user and item characteristics? (3) how to capture and utilize social connections among users? For the first challenge, we propose a novel ranking based matrix factorization model to aggregate explicit and implicit user feedback. For the second challenge, we extend this model to a sequential matrix factorization model to enable time-aware parametrization. Finally, we introduce a network regularization function to constrain user parameters based on social connections. To the best of our knowledge, this is the first study that investigates the collective modeling of social and temporal dynamics. Experiments on the CAMRa 2010 dataset demonstrated clear improvements over many baselines.

The new user problem (aka user cold start) is very common in online recommender systems. Active collaborative filtering (active CF) tries to solve this problem by intelligently soliciting user feedback in order to build an initial user profile with minimal costs. Existing methods only query the user for feedback on items, while users can have preferences over items as well as certain item attributes. In this paper, we extend active CF via user feedback on both items and attributes. For example, when making movie recommendations, the system can ask users for not only their favorite movies, but also attributes such as genres, actors, etc. We design a unified active CF framework for incorporating both item and attribute feedback based on the random walk model. We test the active CF algorithm on real-world movie recommendation data sets to demonstrate that appropriately querying for both item and feature feedback can significantly reduce the overall user effort measured in terms of number of queries. We show that we can achieve much better recommendation quality as compared to traditional active CF methods that support only item feedback.