We propose encoder-centric stepwise models for extractive summarization using structured transformers -HiBERT (Zhang et al., 2019) and Extended Transformers (Ainslie et al., 2020).We enable stepwise summarization by injecting the previously generated summary into the structured transformer as an auxiliary sub-structure.Our models are not only efficient in modeling the structure of long inputs, but they also do not rely on task-specific redundancy-aware modeling, making them a general purpose extractive content planner for different tasks.When evaluated on CNN/DailyMail extractive summarization, stepwise models achieve state-of-the-art performance in terms of Rouge without any redundancy aware modeling or sentence filtering.This also holds true for Rotowire tableto-text generation, where our models surpass previously reported metrics for content selection, planning and ordering, highlighting the strength of stepwise modeling.Amongst the two structured transformers we test, stepwise Extended Transformers provides the best performance across both datasets and sets a new standard for these challenges. 1 * Equal contribution.

Conversational agents offer users a naturallanguage interface to accomplish tasks, entertain themselves, or access information.Informational dialogue is particularly challenging in that the agent has to hold a conversation on an open topic, and to achieve a reasonable coverage it generally needs to digest and present unstructured information from textual sources.Making responses based on such sources sound natural and fit appropriately into the conversation context is a topic of ongoing research, one of the key issues of which is preventing the agent's responses from sounding repetitive.Targeting this issue, we propose a new task, known as redundancy localization, which aims to pinpoint semantic overlap between text passages.To help address it systematically, we formalize the task, prepare a public dataset with fine-grained redundancy labels, and propose a model utilizing a weak training signal defined over the results of a passage-retrieval system on web texts.The proposed model demonstrates superior performance compared to a state-of-the-art entailment model and yields encouraging results when applied to a real-world dialogue. * Work performed during an internship at Google.User: What is Malaria?Agent: A disease caused by a plasmodium parasite, transmitted by the bite of infected mosquitoes.User: Is it a virus?Agent: Malaria is a parasitic infection spread by Anopheles mosquitoes.The Plasmodium parasite that causes Malaria is neither a virus nor a bacterium -it is a single-celled parasite that multiplies in red blood cells of humans as well as in the mosquito intestine.

Guided by multiple heuristics, a unified taxonomy of entities and categories is distilled from the Wikipedia category network. A comprehensive evaluation, based on the analysis of upward generalization paths, demonstrates that the taxonomy supports generalizations which are more than twice as accurate as the state of the art. The taxonomy is available at http://headstaxonomy.com.

In this paper, we experiment with a resource consisting of metaphorically annotated proverbs on the task of word-level metaphor recognition.We observe that existing feature sets do not perform well on this data.We design a novel set of features to better capture the peculiar nature of proverbs and we demonstrate that these new features are significantly more effective on the metaphorically dense proverb data.

Sebastian Krause, Enrique Alfonseca, Katja Filippova, Daniele Pighin. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2015.

Abstractive text summarization of news requires a way of representing events, such as a collection of pattern clusters in which every cluster represents an event (e.g., marriage) and every pattern in the cluster is a way of expressing the event (e.g., X married Y, X and Y tied the knot).We compare three ways of extracting event patterns: heuristics-based, compressionbased and memory-based.While the former has been used previously in multidocument abstraction, the latter two have never been used for this task.Compared with the first two techniques, the memorybased method allows for generating significantly more grammatical and informative sentences, at the cost of searching a vast space of hundreds of millions of parse trees of known grammatical utterances.To this end, we introduce a data structure and a search method that make it possible to efficiently extrapolate from every sentence the parse sub-trees that match against any of the stored utterances.

Supervised approaches to data mining are particularly appealing as they allow for the extraction of complex relations from data objects. In order to facilitate their application in different areas, ranging from protein to protein interaction in bioinformatics to text mining in computational linguistics research, a modular and general mining framework is needed. The major constraint to the generalization process concerns the feature design for the description of relational data. In this paper, we present a machine learning framework for the automatic mining of relations, where the target objects are structurally organized in a tree. Object types are generalized by means of the use of roles, whereas the relation properties are described by means of the underlying tree structure. The latter is encoded in the learning algorithm thanks to kernel methods for structured data, which represent structures in terms of their all possible subparts. This approach can be applied to any kind of data disregarding their very nature. Experiments with support vector machines on two text mining datasets for relation extraction, i.e. the PropBank and FrameNet corpora, show both that our approach is general, and that it reaches state-of-the-art accuracy.

The availability of large scale data sets of manually annotated predicate-argument structures has recently favored the use of machine learning approaches to the design of automated semantic role labeling (SRL) systems. The main research in this area relates to the design choices for feature representation and for effective decompositions of the task in different learning models. Regarding the former choice, structural properties of full syntactic parses are largely employed as they represent ways to encode different principles suggested by the linking theory between syntax and semantics. The latter choice relates to several learning schemes over global views of the parses. For example, re-ranking stages operating over alternative predicate-argument sequences of the same sentence have shown to be very effective. In this article, we propose several kernel functions to model parse tree properties in kernel-based machines, for example, perceptrons or support vector machines. In particular, we define different kinds of tree kernels as general approaches to feature engineering in SRL. Moreover, we extensively experiment with such kernels to investigate their contribution to individual stages of an SRL architecture both in isolation and in combination with other traditional manually coded features. The results for boundary recognition, classification, and re-ranking stages provide systematic evidence about the significant impact of tree kernels on the overall accuracy, especially when the amount of training data is small. As a conclusive result, tree kernels allow for a general and easily portable feature engineering method which is applicable to a large family of natural language processing tasks.