With counterfactual bandit learning, models can be trained based on positive and negative feedback received for historical predictions, with no labeled data needed.Such feedback is often available in real-world dialog systems, however, the modularized architecture commonly used in large-scale systems prevents the direct application of such algorithms.In this paper, we study the feedback attribution problem that arises when using counterfactual bandit learning for multi-domain spoken language understanding.We introduce an experimental setup to simulate the problem on small-scale public datasets, propose attribution methods inspired by multi-agent reinforcement learning and evaluate them against multiple baselines.We find that while directly using overall feedback leads to disastrous performance, our proposed attribution methods can allow training competitive models from user feedback.

While recent progress on abstractive summarization has led to remarkably fluent summaries, factual errors in generated summaries still severely limit their use in practice.In this paper, we evaluate summaries produced by state-of-the-art models via crowdsourcing and show that such errors occur frequently, in particular with more abstractive models.We study whether textual entailment predictions can be used to detect such errors and if they can be reduced by reranking alternative predicted summaries.That leads to an interesting downstream application for entailment models.In our experiments, we find that outof-the-box entailment models trained on NLI datasets do not yet offer the desired performance for the downstream task and we therefore release our annotations as additional test data for future extrinsic evaluations of NLI.

Concept maps can be used to concisely represent important information and bring structure into large document collections.Therefore, we study a variant of multidocument summarization that produces summaries in the form of concept maps.However, suitable evaluation datasets for this task are currently missing.To close this gap, we present a newly created corpus of concept maps that summarize heterogeneous collections of web documents on educational topics.It was created using a novel crowdsourcing approach that allows us to efficiently determine important elements in large document collections.We release the corpus along with a baseline system and proposed evaluation protocol to enable further research on this variant of summarization. 1

Concept-map-based multi-document summarization is a variant of traditional summarization that produces structured summaries in the form of concept maps. In this work, we propose a new model for the task that addresses several issues in previous methods. It learns to identify and merge coreferent concepts to reduce redundancy, determines their importance with a strong supervised model and finds an optimal summary concept map via integer linear programming. It is also computationally more efficient than previous methods, allowing us to summarize larger document sets. We evaluate the model on two datasets, finding that it outperforms several approaches from previous work.

Many downstream NLP tasks can benefit from Open Information Extraction (Open IE) as a semantic representation.While Open IE systems are available for English, many other languages lack such tools.In this paper, we present a straightforward approach for adapting PropS, a rule-based predicate-argument analysis for English, to a new language, German.With this approach, we quickly obtain an Open IE system for German covering 89% of the English rule set.It yields 1.6 n-ary extractions per sentence at 60% precision, making it comparable to systems for English and readily usable in downstream applications.1