Naturally occurring information-seeking questions often contain questionable assumptions-assumptions that are false or unverifiable.Questions containing questionable assumptions are challenging because they require a distinct answer strategy that deviates from typical answers for information-seeking questions.For instance, the question When did Marie Curie discover Uranium?cannot be answered as a typical when question without addressing the false assumption Marie Curie discovered Uranium.In this work, we propose (QA) 2 (Question Answering with Questionable Assumptions), an open-domain evaluation dataset consisting of naturally occurring search engine queries that may or may not contain questionable assumptions.To be successful on (QA) 2 , systems must be able to detect questionable assumptions and also be able to produce adequate responses for both typical information-seeking questions and ones with questionable assumptions.Through human rater acceptability on end-to-end QA with (QA) 2 , we find that current models do struggle with handling questionable assumptions, leaving substantial headroom for progress.* Equal contribution, corresponding authors ∆ Work partly done at NYU before joining BU. δ Work done at NYU before joining Amazon. 1 We use the term questionable assumptions instead of presupposition failure to capture failures of both true presup-

Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Samuel R. Bowman. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, Samuel R. Bowman. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, Samuel R. Bowman. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

A substantial thread of recent work on latent tree learning has attempted to develop neural network models with parse-valued latent variables and train them on non-parsing tasks, in the hope of having them discover interpretable tree structure.In a recent paper, Shen et al. (2018) introduce such a model and report nearstate-of-the-art results on the target task of language modeling, and the first strong latent tree learning result on constituency parsing.In an attempt to reproduce these results, we discover issues that make the original results hard to trust, including tuning and even training on what is effectively the test set.Here, we attempt to reproduce these results in a fair experiment and to extend them to two new datasets.We find that the results of this work are robust: All variants of the model under study outperform all latent tree learning baselines, and perform competitively with symbolic grammar induction systems.We find that this model represents the first empirical success for latent tree learning, and that neural network language modeling warrants further study as a setting for grammar induction.