Abstract Since the SciCap dataset’s launch in 2021, the research community has made significant progress in generating captions for scientific figures in scholarly articles. In 2023, the first SciCap Challenge took place, inviting global teams to use an expanded SciCap dataset to develop models for captioning diverse figure types across various academic fields. At the same time, text generation models advanced quickly, with many powerful pre-trained large multimodal models (LMMs) emerging that showed impressive capabilities in various vision-and-language tasks. This paper presents an overview of the first SciCap Challenge and details the performance of various models on its data, capturing a snapshot of the field’s state. We found that professional editors overwhelmingly preferred figure captions generated by GPT-4V over those from all other models and even the original captions written by authors. Following this key finding, we conducted detailed analyses to answer this question: Have advanced LMMs solved the task of generating captions for scientific figures?

Recent studies indicated GPT-4 outperforms online crowd workers in data labeling accuracy, notably workers from Amazon Mechanical Turk (MTurk). However, these studies were criticized for deviating from standard crowdsourcing practices and emphasizing individual workers’ performances over the whole data-annotation process. This paper compared GPT-4 and an ethical and well-executed MTurk pipeline, with 415 workers labeling 3,177 sentence segments from 200 scholarly articles using the CODA-19 scheme. Two worker interfaces yielded 127,080 labels, which were then used to infer the final labels through eight label-aggregation algorithms. Our evaluation showed that despite best practices, MTurk pipeline’s highest accuracy was 81.5%, whereas GPT-4 achieved 83.6%. Interestingly, when combining GPT-4’s labels with crowd labels collected via an advanced worker interface for aggregation, 2 out of the 8 algorithms achieved an even higher accuracy (87.5%, 87.0%). Further analysis suggested that, when the crowd’s and GPT-4’s labeling strengths are complementary, aggregating them could increase labeling accuracy.

Chieh-Yang Huang, Ting-Yao Hsu, Ryan Rossi, Ani Nenkova, Sungchul Kim, Gromit Yeuk-Yin Chan, Eunyee Koh, C Lee Giles, Ting-Hao Huang. Proceedings of the 16th International Natural Language Generation Conference. 2023.

This paper introduces semantic frame forecast, a task that predicts the semantic frames that will occur in the next 10, 100, or even 1,000 sentences in a running story.Prior work focused on predicting the immediate future of a story, such as one to a few sentences ahead.However, when novelists write long stories, generating a few sentences is not enough to help them gain high-level insight to develop the follow-up story.In this paper, we formulate a long story as a sequence of "story blocks," where each block contains a fixed number of sentences (e.g., 10, 100, or 200).This formulation allows us to predict the follow-up story arc beyond the scope of a few sentences.We represent a story block using the term frequencies (TF) of semantic frames in it, normalized by each frame's inverse document frequency (IDF).We conduct semantic frame forecast experiments on 4,794 books from the Bookcorpus and 7,962 scientific abstracts from CODA-19, with block sizes ranging from 5 to 1,000 sentences.The results show that automated models can forecast the follow-up story blocks better than the random, prior, and replay baselines, indicating the task's feasibility.We also learn that the models using the frame representation as features outperform all the existing approaches when the block size is over 150 sentences.The human evaluation also shows that the proposed frame representation, when visualized as word clouds, is comprehensible, representative, and specific to humans.Our code is available at: https://github.com/ appleternity/FrameForecasting.

Ideation is essential for creative writing. Many authors struggle to come up with ideas throughout the writing process, yet modern writing tools fail to provide on-the-spot assistance for writers when they get stuck. This paper introduces Heteroglossia, an add-on for Google Docs that allows writers to elicit story ideas from the online crowd using their text editors. Writers can share snippets of their working drafts and ask the crowd to provide follow-up story ideas based on it. Heteroglossia employs a strategy called "role play", where each worker is assigned a fictional character in a story and asked to brainstorm plot ideas from that character's perspective. Our deployment with two experienced story writers shows that Heteroglossia is easy to use and can generate interesting ideas. Heteroglossia allows us to gain insight into how future technologies can be developed to support ideation in creative writing.

Many English-as-a-second language learners have trouble using near-synonym words (e.g., small vs.little; briefly vs.shortly) correctly, and often look for example sentences to learn how two nearly synonymous terms differ. Prior work uses hand-crafted scores to recommend sentences but has difficulty in adopting such scores to all the near-synonyms as near-synonyms differ in various ways. We notice that the helpfulness of the learning material would reflect on the learners’ performance. Thus, we propose the inference-based learner-like agent to mimic learner behavior and identify good learning materials by examining the agent’s performance. To enable the agent to behave like a learner, we leverage entailment modeling’s capability of inferring answers from the provided materials. Experimental results show that the proposed agent is equipped with good learner-like behavior to achieve the best performance in both fill-in-the-blank (FITB) and good example sentence selection tasks. We further conduct a classroom user study with college ESL learners. The results of the user study show that the proposed agent can find out example sentences that help students learn more easily and efficiently. Compared to other models, the proposed agent improves the score of more than 17% of students after learning.

We introduce the first dataset for human edits of machine-generated visual stories and explore how these collected edits may be used for the visual story post-editing task.The dataset, VIST-Edit 1 , includes 14,905 humanedited versions of 2,981 machine-generated visual stories.The stories were generated by two state-of-the-art visual storytelling models, each aligned to 5 human-edited versions.We establish baselines for the task, showing how a relatively small set of human edits can be leveraged to boost the performance of large visual storytelling models.We also discuss the weak correlation between automatic evaluation scores and human ratings, motivating the need for new automatic metrics.

Recently chatbots have been widely deployed in social network messengers. To make these chatbots more human, we present EmotionPush, the first social spoken-language private dialog dataset containing instant message logs and the unique read event logs, containing a total of 162,031 message logs and their corresponding read logs of real private conversations on Facebook Messenger. We ensure data privacy by masking all the named entities with a code composed by their type and an unique ID. In addition, we take care of the debug need by releasing messages partially by their original words and partially in the form of word embeddings. In addition, with this dataset, we propose the emotion classification task and a novel response-time prediction task to enhance the humanity of chatbots. We establish strong baselines for these two tasks. Experiment results show that EmotionPush helps to achieve over 90% accuracy for major emotion classification and 89% accuracy for response time prediction. We expect to enable chatbots to know when to respond and what message to send to encourage user responses.

The rapid growth of web source has changed language learning behavior. More and more people utilized web sources instead of paper books. However, the problem now is that it is overwhelming to find useful information. In addition, when considering using different words, good example sentences demonstrating nuance among words are extremely helpful but learners can hardly find them as most web dictionaries contain explanations and examples for only a single word. To solve the problem, we proposed a system called GiveMeExample which can automatically search for the best example sentences of a group of confusing words. The proposed system learns the word usage model for each word in the confusing word group and a universal difficulty model for all sentences, in order to propose simple but clear example sentences for learners. Experiments show that the proposed approach can really provide the most useful example sentences for understanding confusing words. GiveMeExample is available at http://givemeexample.com/GiveMeExample.