An interesting behavior in large language models (LLMs) is prompt sensitivity. When provided with different but semantically equivalent versions of the same prompt, models may produce very different distributions of answers. This suggests that the uncertainty reflected in a model's output distribution for one prompt may not reflect the model's uncertainty about the meaning of the prompt. We model prompt sensitivity as a type of generalization error, and show that sampling across the semantic concept space with paraphrasing perturbations improves uncertainty calibration without compromising accuracy. Additionally, we introduce a new metric for uncertainty decomposition in black-box LLMs that improves upon entropy-based decomposition by modeling semantic continuities in natural language generation. We show that this decomposition metric can be used to quantify how much LLM uncertainty is attributed to prompt sensitivity. Our work introduces a new way to improve uncertainty calibration in prompt-sensitive language models, and provides evidence that some LLMs fail to exhibit consistent general reasoning about the meanings of their inputs.

Chi-Yang Hsu, Yun-Wei Chu, Vincent Chen, Kuan-Chieh Lo, Chacha Chen, Ting-Hao Huang, Lun-Wei Ku. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

Stories are diverse and highly personalized, resulting in a large possible output space for story generation. Existing end-to-end approaches produce monotonous stories because they are limited to the vocabulary and knowledge in a single training dataset. This paper introduces KG-Story, a three-stage framework that allows the story generation model to take advantage of external Knowledge Graphs to produce interesting stories. KG-Story distills a set of representative words from the input prompts, enriches the word set by using external knowledge graphs, and finally generates stories based on the enriched word set. This distill-enrich-generate framework allows the use of external resources not only for the enrichment phase, but also for the distillation and generation phases. In this paper, we show the superiority of KG-Story for visual storytelling, where the input prompt is a sequence of five photos and the output is a short story. Per the human ranking evaluation, stories generated by KG-Story are on average ranked better than that of the state-of-the-art systems. Our code and output stories are available at https://github.com/zychen423/KE-VIST.

Ride-sharing platforms have rapidly spread and disrupted ride hailing markets, resulting in conflicts between ride-sharing and taxi drivers. Taxi drivers claim that their counterparts have unfair advantages in terms of lower prices and a more stable customer base, making it difficult to earn a living. Local government entities have dealt with this disruption and conflict in different ways, often looking towards some form of regulation. While there have been discussions about what the regulation should be, there has been less work looking at what impacts regulations have on ride-sharing drivers and their usage of the platforms. In this paper we present our interview study of ride-sharing drivers in Taiwan, who have gone through three distinct phases of regulation. Drivers felt that regulations legitimized their work, while having to navigate consequences related to regulated access to platforms and fundamental changes to the "gig'' of ride-sharing.

Previously, the research team designed a game-based writing environment to support students’ writing development. However, students are not good at peer response. It decreased the effect of talking for revising strategy. Therefore, the purpose of this study designed writing mini-lesson and enhanced students’ skill of giving peer response and revising their draft based on the received response. A quasi-experiment was conducted and 108 three-grade students from 4 classes participated in. The experimental group received the three writing mini-lessons during the semester and the control group received general instruction. Both of these two group students wrote three writing article on game-based writing environment. The writing minilesson was implemented in the beginning of every writing course and cost around 5-8 minutes. The results showed that there is no significant different on the number of peer response between control and experimental group. In contrast, the type of peer response was different between these two groups. It implied that writing mini-lesson could provide students how to give concrete response and proved the effect of writing mini-lesson for elementary school students.