Generating coherent long texts is an important yet challenging task, particularly for the openended generation task.Prior work based on discrete latent codes focuses on the modeling of discourse relation, resulting in discrete codes only learning shallow semantics (Ji and Huang, 2021).A natural text always revolves around several related topics and the transition across them is natural and smooth.In this work, we investigate whether discrete latent codes can learn information of topics.To this end, we build a topic-aware latent code-guided text generation model.To encourage discrete codes to model information about topics, we propose a span-level bag-of-words training objective for the model.Automatic and manual evaluation experiments show that our method can generate more topic-relevant and coherent texts.

Previous works on syntactically controlled paraphrase generation heavily rely on largescale parallel paraphrase data that are not easily available for many languages and domains.In this paper, we take this research direction to the extreme and investigate whether it is possible to learn syntactically controlled paraphrase generation with non-parallel data.We propose a syntactically-informed unsupervised paraphrasing model based on conditional variational auto-encoder (VAE) which can generate texts in a specified syntactic structure.Particularly, we design a two-stage learning method to effectively train the model using non-parallel data.The conditional VAE is trained to reconstruct the input sentence according to the given input and its syntactic structure.Furthermore, to improve the syntactic controllability and semantic consistency of the pre-trained conditional VAE, we finetune it using syntax controlling and cycle reconstruction learning objectives, and employ Gumbel-Softmax to combine these new learning objectives.Experiment results demonstrate that the proposed model trained only on non-parallel data is capable of generating diverse paraphrases with specified structures.Additionally, we further validate the effectiveness of our method for generating syntactically adversarial examples on a sentiment analysis task.

Paraphrase generation (PG) is of great importance to many downstream tasks in natural language processing.Diversity is an essential nature to PG for enhancing generalization capability and robustness of downstream applications.Recently, neural sequence-to-sequence (Seq2Seq) models have shown promising results in PG.However, traditional model training for PG focuses on optimizing model prediction against single reference and employs cross-entropy loss, which objective is unable to encourage model to generate diverse paraphrases.In this work, we present a novel approach with multi-objective learning to PG.We propose a learning-exploring method to generate sentences as learning objectives from the learned data distribution, and employ reinforcement learning to combine these new learning objectives for model training.We first design a sample-based algorithm to explore diverse sentences.Then we introduce several reward functions to evaluate the sampled sentences as learning signals in terms of expressive diversity and semantic fidelity, aiming to generate diverse and high-quality paraphrases.To effectively optimize model performance satisfying different evaluating aspects, we use a GradNorm-based algorithm that automatically balances these training objectives.Experiments and analyses on Quora and Twitter datasets demonstrate that our proposed method not only gains a significant increase in diversity but also improves generation quality over several state-of-the-art baselines.

Paraphrase generation is of great importance to many downstream tasks in natural language processing. Recent efforts have focused on generating paraphrases in specific syntactic forms, which, generally, heavily relies on manually annotated paraphrase data that is not easily available for many languages and domains. In this paper, we propose a novel end-to-end framework to leverage existing large-scale bilingual parallel corpora to generate paraphrases under the control of syntactic exemplars. In order to train one model over the two languages of parallel corpora, we embed sentences of them into the same content and style spaces with shared content and style encoders using cross-lingual word embeddings. We propose an adversarial discriminator to disentangle the content and style space, and employ a latent variable to model the syntactic style of a given exemplar in order to guide the two decoders for generation. Additionally, we introduce cycle and masking learning schemes to efficiently train the model. Experiments and analyses demonstrate that the proposed model trained only on bilingual parallel data is capable of generating diverse paraphrases with desirable syntactic styles. Fine-tuning the trained model on a small paraphrase corpus makes it substantially outperform state-of-the-art paraphrase generation models trained on a larger paraphrase dataset.