We propose a method to optimize language model pre-training data mixtures through efficient approximation of the cross-entropy loss corresponding to each candidate mixture via a Mixture of Data Experts (MDE).We use this approximation as additional features in a regression model, trained from observations of model loss for a small number of mixtures.Experiments with Transformer language models between 70M and 10B parameters on the SlimPajama dataset show that our method achieves significantly better performance than approaches that train regression models using only the mixture rates as input features.Combining our method with an objective that takes into account cross-entropy on end task data leads to superior performance on few-shot downstream evaluations.We also provide theoretical insights on why aggregation of data expert predictions can provide good approximations to model losses for data mixtures.

Gaze estimation is a crucial component of human-robot Interaction (HRI). While previous gaze estimation methods have been widely applied in advertising and gaming with head-mounted devices or complicated camera systems, little research has been conducted in everyday HRI scenarios. During interactions, robots have the potential to infer human intention through static gaze directions and dynamic eye movements, enabling them to behave more intelligently and friendly. This paper combines appearance-based gaze estimation methods with eye movement analysis methods to infer human intentions, particularly in human-robot interaction scenarios. Real interactions were conducted to test the accuracy and robustness of the methods developed. The experiments demonstrate that our methods deliver practical, robust, and accurate results.

Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, Kristina Toutanova. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

Tianze Shi, Lillian Lee. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Tianze Shi, Adrian Benton, Igor Malioutov, Ozan İrsoy. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Tianze Shi, Ozan İrsoy, Igor Malioutov, Lillian Lee. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

An interesting and frequent type of multiword expression (MWE) is the headless MWE, for which there are no true internal syntactic dominance relations; examples include many named entities ("Wells Fargo") and dates ("July 5, 2020") as well as certain productive constructions ("blow for blow", "day after day").Despite their special status and prevalence, current dependency-annotation schemes require treating such flat structures as if they had internal syntactic heads, and most current parsers handle them in the same fashion as headed constructions.Meanwhile, outside the context of parsing, taggers are typically used for identifying MWEs, but taggers might benefit from structural information.We empirically compare these two common strategies-parsing and tagging-for predicting flat MWEs.Additionally, we propose an efficient joint decoding algorithm that combines scores from both strategies.Experimental results on the MWE-Aware English Dependency Corpus and on six non-English dependency treebanks with frequent flat structures show that: (1) tagging is more accurate than parsing for identifying flat-structure MWEs, (2) our joint decoder reconciles the two different views and, for non-BERT features, leads to higher accuracies, and (3) most of the gains result from feature sharing between the parsers and taggers.

We reduce the task of (span-based) PropBankstyle semantic role labeling (SRL) to syntactic dependency parsing.Our approach is motivated by our empirical analysis that shows three common syntactic patterns account for over 98% of the SRL annotations for both English and Chinese data.Based on this observation, we present a conversion scheme that packs SRL annotations into dependency tree representations through joint labels that permit highly accurate recovery back to the original format.This representation allows us to train statistical dependency parsers to tackle SRL and achieve competitive performance with the current state of the art.Our findings show the promise of syntactic dependency trees in encoding semantic role relations within their syntactic domain of locality, and point to potential further integration of syntactic methods into semantic role labeling in the future.

Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, Dragomir Radev. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Rui Zhang, Tao Yu, Heyang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, Dragomir Radev. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Shi, Huang, and Lee (2017a) obtained state-of-the-art results for English and Chinese dependency parsing by combining dynamic-programming implementations of transition-based dependency parsers with a minimal set of bidirectional LSTM features.However, their results were limited to projective parsing.In this paper, we extend their approach to support non-projectivity by providing the first practical implementation of the MH 4 algorithm, an Opn 4 q mildly nonprojective dynamic-programming parser with very high coverage on non-projective treebanks.To make MH 4 compatible with minimal transition-based feature sets, we introduce a transition-based interpretation of it in which parser items are mapped to sequences of transitions.We thus obtain the first implementation of global decoding for non-projective transition-based parsing, and demonstrate empirically that it is more effective than its projective counterpart in parsing a number of highly non-projective languages.

We present a complete, automated, and efficient approach for utilizing valency analysis in making dependency parsing decisions.It includes extraction of valency patterns, a probabilistic model for tagging these patterns, and a joint decoding process that explicitly considers the number and types of each token's syntactic dependents.On 53 treebanks representing 41 languages in the Universal Dependencies data, we find that incorporating valency information yields higher precision and F1 scores on the core arguments (subjects and complements) and functional relations (e.g., auxiliaries) that we employ for valency analysis.Precision on core arguments improves from 80.87 to 85.43.We further show that our approach can be applied to an ostensibly different formalism and dataset, Tree Adjoining Grammar as extracted from the Penn Treebank; there, we outperform the previous stateof-the-art labeled attachment score by 0.7.Finally, we explore the potential of extending valency patterns beyond their traditional domain by confirming their helpfulness in improving PP attachment decisions.1

We first present a minimal feature set for transition-based dependency parsing, continuing a recent trend started by Kiperwasser and Goldberg (2016a) and Cross and Huang (2016a) of using bi-directional LSTM features.We plug our minimal feature set into the dynamic-programming framework of Huang and Sagae (2010) and Kuhlmann et al. (2011) to produce the first implementation of worst-case Opn 3 q exact decoders for arc-hybrid and arceager transition systems.With our minimal features, we also present Opn 3 q global training methods.Finally, using ensembles including our new parsers, we achieve the best unlabeled attachment score reported (to our knowledge) on the Chinese Treebank and the "second-best-in-class" result on the English Penn Treebank.