Yun-Shiuan Chuang, Sameer Narendran, Nikunj Harlalka, Alexander Cheung, Sizhe Gao, Siddharth Suresh, Junjie Hu, Timothy T. Rogers. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Robot-human trust is an important concern as robots become integrated into human spaces. We tested a method grounded in psychological theory to increase human-robot trust—synchronous motion. Human participants completed a goal-oriented ball-moving task with a robotic arm to sound cues that were synchronous or asynchronous with the robot’s pacing. Participants were instructed to follow sound cues without information about synchrony. We found that participants in the synchrony condition trusted the robot to complete a new task that was comparable to the task they completed, significantly more than the asynchrony condition. However, this effect did not extend to harder tasks. The participants in the synchrony condition also believed that the robot had more influence on the outcomes of the new task compared to the asynchrony condition. On average, participants’ trust increased with the robotic arm after completing the task, regardless of condition. We report findings from a thematic analysis that demonstrate that participants in the synchrony condition found synchrony to be beneficial, while participants in the asynchrony condition found it cognitively taxing to be out-of-sync. Results from this work may be used to improve human-robot interactions in various contexts.

We present a novel multimodal preference dataset for creative tasks, consisting of over 250 million human votes on more than 2.2 million captions, collected through crowdsourcing rating data for The New Yorker's weekly cartoon caption contest over the past eight years. This unique dataset supports the development and evaluation of multimodal large language models and preference-based fine-tuning algorithms for humorous caption generation. We propose novel benchmarks for judging the quality of model-generated captions, utilizing both GPT4 and human judgments to establish ranking-based evaluation strategies. Our experimental results highlight the limitations of current fine-tuning methods, such as RLHF and DPO, when applied to creative tasks. Furthermore, we demonstrate that even state-of-the-art models like GPT4 and Claude currently underperform top human contestants in generating humorous captions. As we conclude this extensive data collection effort, we release the entire preference dataset to the research community, fostering further advancements in AI humor generation and evaluation.

Neural network models of language have long been used as a tool for developing hypotheses about conceptual representation in the mind and brain.For many years, such use involved extracting vector-space representations of words and using distances among these to predict or understand human behavior in various semantic tasks.Contemporary large language models (LLMs), however, make it possible to interrogate the latent structure of conceptual representations using experimental methods nearly identical to those commonly used with human participants.The current work utilizes three common techniques borrowed from cognitive psychology to estimate and compare the structure of concepts in humans and a suite of LLMs.In humans, we show that conceptual structure is robust to differences in culture, language, and method of estimation.Structures estimated from LLM behavior, while individually fairly consistent with those estimated from human behavior, vary much more depending upon the particular task used to generate responsesacross tasks, estimates of conceptual structure from the very same model cohere less with one another than do human structure estimates.These results highlight an important difference between contemporary LLMs and human cognition, with implications for understanding some fundamental limitations of contemporary machine language.