Francine Chen, Scott Carter, Tatiana Lau, Nayeli Suseth Bravo, Sumanta Bhattacharyya, Kate Sieck, Charlene C. Wu. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

The Wizard of Oz (WoZ) method is a widely adopted research approach where a human Wizard “role-plays” a not readily available technology and interacts with participants to elicit user behaviors and probe the design space. With the growing ability for modern large language models (LLMs) to role-play, one can apply LLMs as Wizards in WoZ experiments with better scalability and lower cost than the traditional approach. However, methodological guidance on responsibly applying LLMs in WoZ experiments and a systematic evaluation of LLMs’ role-playing ability are lacking. Through two LLM-powered WoZ studies, we take the first step towards identifying an experiment lifecycle for researchers to safely integrate LLMs into WoZ experiments and interpret data generated from settings that involve Wizards role-played by LLMs. We also contribute a heuristic-based evaluation framework that allows the estimation of LLMs’ role-playing ability in WoZ experiments and reveals LLMs’ behavior patterns at scale.

With the rising popularity of electrified vehicles, emphasis has been placed on encouraging charging with renewable energy and maximizing battery longevity to improve vehicle sustainability. Many mobile applications offer tools to suggest charging times with more sustainable renewable energy and charging strategies that preserve battery health. However, these options often result in longer, less convenient charging times for drivers. Here we conducted three charging scenario studies to identify factors that influence willingness to wait for sustainable charging. Participants selected between faster but less sustainable charging options and slower charging options that either reduce charging emissions or improve battery longevity. We find people’s willingness to wait for green energy is influenced by situational factors; further we find that information and battery longevity interventions can increase willingness to wait for sustainable charging. Finally, we provide design recommendations to promote sustainably in charging behaviors.

Narrative story generation has gained emerging interest in the field of large language models. The present paper aims to compare stories generated by an LLM only (non-interleaved) with those generated by interleaving human-generated and LLM-generated text (interleaved). The study’s hypothesis is that interleaved stories would perform better than non-interleaved stories. To verify this hypothesis, we conducted two tests with roughly 500 participants each. Participants were asked to rate stories of each type, including an overall score or preference and four facets—logical soundness, plausibility, understandability, and novelty. Our findings indicate that interleaved stories were in fact less preferred than non-interleaved stories. The result has implications for the design and implementation of our story generators. This study contributes new insights into the potential uses and restrictions of interleaved and non-interleaved systems regarding generating narrative stories, which may help to improve the performance of such story generators.

Electrification is an important first step toward reducing the greenhouse emissions of passenger vehicles. However, how drivers drive, charge, and operate their electrified vehicles can have a large impact on their emissions, particularly for Plug-in Hybrid Electric vehicles (PHEVs) that combine all-electric driving with an internal combustion engine. In this paper, we investigate how and why drivers use their PHEVs and uncover design opportunities for interfaces that can support the efficient use of PHEVs. We used a mixed-method approach combining quantitative, qualitative, and concept elicitation methods with PHEV owners in the US. While past findings indicate that PHEV drivers are not motivated to charge regularly, our work contradicts this with evidence of (1) regular charging with home infrastructure, (2) high cost sensitivity, and (3) preference for driving in all-electric mode. Our results indicate that the most critical problem is inadequate user support for navigating poor charging infrastructure.

From ride-hailing to car rentals, consumers are often presented with eco-friendly options. Beyond highlighting a “green” vehicle and CO2 emissions, CO2 equivalencies have been designed to provide understandable amounts; we ask which equivalencies will lead to eco-friendly decisions. We conducted five ride-hailing scenario surveys where participants picked between regular and eco-friendly options, testing equivalencies, social features, and valence-based interventions. Further, we tested a car-rental embodiment to gauge how an individual (needing a car for several days) might behave versus the immediate ride-hailing context. We find that participants are more likely to choose green rides when presented with additional information about emissions; CO2 by weight was found to be the most effective. Further, we found that information framing—be it individual or collective footprint, positive or negative valence—had an impact on participants’ choices. Finally, we discuss how our findings inform the design of effective interventions for reducing car-based carbon-emissions.