Text generation often involves producing texts that also satisfy a given set of semantic constraints.While most approaches for conditional text generation have primarily focused on lexical constraints, they often struggle to effectively incorporate syntactic constraints, which provide a richer language for approximating semantic constraints.We address this gap by introducing NEUROSTRUCTURAL DECODING, a new decoding algorithm that incorporates syntactic constraints to further improve the quality of the generated text.We build NEUROSTRUC-TURAL DECODING on the NeuroLogic Decoding (Lu et al., 2021b) algorithm, which enables language generation models to produce fluent text while satisfying complex lexical constraints.Our algorithm is powerful and scalable.It tracks lexico-syntactic constraints (e.g., we need to observe dog as subject and ball as object) during decoding by parsing the partial generations at each step.To this end, we adapt a dependency parser to generate parses for incomplete sentences.Our approach is evaluated on three different language generation tasks, and the results show improved performance in both lexical and syntactic metrics compared to previous methods.The results suggest this is a promising solution for integrating fine-grained controllable text generation into the conventional beam search decoding 1 .

Can language models read biomedical texts and explain the biomedical mechanisms discussed? In this work we introduce a biomedical mechanism summarization task. Biomedical studies often investigate the mechanisms behind how one entity (e.g., a protein or a chemical) affects another in a biological context. The abstracts of these publications often include a focused set of sentences that present relevant supporting statements regarding such relationships, associated experimental evidence, and a concluding sentence that summarizes the mechanism underlying the relationship. We leverage this structure and create a summarization task, where the input is a collection of sentences and the main entities in an abstract, and the output includes the relationship and a sentence that summarizes the mechanism. Using a small amount of manually labeled mechanism sentences, we train a mechanism sentence classifier to filter a large biomedical abstract collection and create a summarization dataset with 22k instances. We also introduce conclusion sentence generation as a pretraining task with 611k instances. We benchmark the performance of large bio-domain language models. We find that while the pretraining task help improves performance, the best model produces acceptable mechanism outputs in only 32% of the instances, which shows the task presents significant challenges in biomedical language understanding and summarization.

Predicting how events induce emotions in the characters of a story is typically seen as a standard multi-label classification task, which usually treats labels as anonymous classes to predict.They ignore information that may be conveyed by the emotion labels themselves.We propose that the semantics of emotion labels can guide a model's attention when representing the input story.Further, we observe that the emotions evoked by an event are often related: an event that evokes joy is unlikely to also evoke sadness.In this work, we explicitly model label classes via label embeddings, and add mechanisms that track label-label correlations both during training and inference.We also introduce a new semi-supervision strategy that regularizes for the correlations on unlabeled data.Our empirical evaluations show that modeling label semantics yields consistent benefits, and we advance the state-of-theart on an emotion inference task.

We introduce PerSenT, a dataset of crowd-sourced annotations of the sentiment expressed by the authors towards the main entities in news articles. The dataset also includes paragraph-level sentiment annotations to provide more fine-grained supervision for the task. Our benchmarks of multiple strong baselines show that this is a difficult classification task. The results also suggest that simply fine-tuning document-level representations from BERT isn't adequate for this task. Making paragraph-level decisions and aggregating them over the entire document is also ineffective. We present empirical and qualitative analyses that illustrate the specific challenges posed by this dataset. We release this dataset with 5.3k documents and 38k paragraphs covering 3.2k unique entities as a challenge in entity sentiment analysis.