Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, Yuwei Fang. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Recent advances in commonsense reasoning have been fueled by the availability of large-scale human annotated datasets. Manual annotation of such datasets, many of which are based on existing knowledge bases, is expensive and not scalable. Moreover, it is challenging to build augmentation data for commonsense reasoning because the synthetic questions need to adhere to real-world scenarios. Hence, we present GraDA, a graph-generative data augmentation framework to synthesize factual data samples from knowledge graphs for commonsense reasoning datasets. First, we train a graph-to-text model for conditional generation of questions from graph entities and relations. Then, we train a generator with GAN loss to generate distractors for synthetic questions. Our approach improves performance for SocialIQA, CODAH, HellaSwag and CommonsenseQA, and works well for generative tasks like ProtoQA. We show improvement in robustness to semantic adversaries after training with GraDA and provide human evaluation of the quality of synthetic datasets in terms of factuality and answerability. Our work provides evidence and encourages future research into graph-based generative data augmentation.

Commonsense reasoning tasks follow a standard paradigm of finetuning pretrained language models on the target task data, where samples are introduced to the model in a random order during training.However, recent research suggests that data order can have a significant impact on the performance of finetuned models for natural language understanding.Hence, we examine the effect of a humanlike easy-to-difficult curriculum during finetuning of language models for commonsense reasoning tasks.We use paced curriculum learning to rank data and sample training mini-batches with increasing levels of difficulty from the ranked dataset during finetuning.Further, we investigate the effect of an adaptive curriculum, i.e., the data ranking is dynamically updated during training based on the current state of the learner model.We use a teacher model to measure difficulty of each sample and experiment with three measures based on question answering probability, variability and out-ofdistribution.To understand the effectiveness of curriculum learning in various scenarios, we apply it on full model fine-tuning as well as parameter-efficient prompt-tuning settings.Our results show that fixed as well as adaptive curriculum learning significantly improve performance for five commonsense reasoning tasks, i.e., SocialIQA, CosmosQA, CODAH, HellaSwag, WinoGrande in both tuning settings.Further, we find that prioritizing the difficult samples in the tail end of training improves generalization to unseen in-domain data as well as out-of-domain data.Our work provides evidence and encourages research into curriculum learning for commonsense reasoning.1 .

While much research has been done in textto-image synthesis, little work has been done to explore the usage of linguistic structure of the input text.Such information is even more important for story visualization since its inputs have an explicit narrative structure that needs to be translated into an image sequence (or visual story).Prior work in this domain has shown that there is ample room for improvement in the generated image sequence in terms of visual quality, consistency and relevance.In this paper, we first explore the use of constituency parse trees using a Transformer-based recurrent architecture for encoding structured input.Second, we augment the structured input with commonsense information and study the impact of this external knowledge on the generation of visual story.Third, we also incorporate visual structure via bounding boxes and dense captioning to provide feedback about the characters/objects in generated images within a dual learning setup.We show that off-theshelf dense-captioning models trained on Visual Genome can improve the spatial structure of images from a different target domain without needing fine-tuning.We train the model end-to-end using intra-story contrastive loss (between words and image sub-regions) and show significant improvements in visual quality.Finally, we provide an analysis of the linguistic and visuo-spatial information.1 1 Code and

Story visualization is an underexplored task that falls at the intersection of many important research directions in both computer vision and natural language processing.In this task, given a series of natural language captions which compose a story, an agent must generate a sequence of images that correspond to the captions.Prior work has introduced recurrent generative models which outperform text-to-image synthesis models on this task.However, there is room for improvement of generated images in terms of visual quality, coherence and relevance.We present a number of improvements to prior modeling approaches, including (1) the addition of a dual learning framework that utilizes video captioning to reinforce the semantic alignment between the story and generated images, (2) a copy-transform mechanism for sequentiallyconsistent story visualization, and (3) MARTbased transformers to model complex interactions between frames.We present ablation studies to demonstrate the effect of each of these techniques on the generative power of the model for both individual images as well as the entire narrative.Furthermore, due to the complexity and generative nature of the task, standard evaluation metrics do not accurately reflect performance.Therefore, we also provide an exploration of evaluation metrics for the model, focused on aspects of the generated frames such as the presence/quality of generated characters, the relevance to captions, and the diversity of the generated images.We also present correlation experiments of our proposed automated metrics with human evaluations.1