Simulating learner actions helps stress-test open-ended interactive learning environments and prototype new adaptations before deployment. While recent studies show the promise of using large language models (LLMs) for simulating human behavior, such approaches have not gone beyond rudimentary proof-of-concept stages due to key limitations. First, LLMs are highly sensitive to minor prompt variations, raising doubts about their ability to generalize to new scenarios without extensive prompt engineering. Moreover, apparently successful outcomes can often be unreliable, either because domain experts unintentionally guide LLMs to produce expected results, leading to self-fulfilling prophecies; or because the LLM has encountered highly similar scenarios in its training data, meaning that models may not be simulating behavior so much as regurgitating memorized content. To address these challenges, we propose Hyp-Mix, a simulation authoring framework that allows experts to develop and evaluate simulations by combining testable hypotheses about learner behavior. Testing this framework in a physics learning environment, we found that GPT-4 Turbo maintains calibrated behavior even as the underlying learner model changes, providing the first evidence that LLMs can be used to simulate realistic behaviors in open-ended interactive learning environments, a necessary prerequisite for useful LLM behavioral simulation.

There has been recent interest in the design of collaborative learning activities that are distributed across multiple technology devices for students to engage in scientific inquiry. Emerging research has begun to investigate students’ collaborative behaviors across different device types and students’ shared attention by tracking eye gaze, body posture, and their interactions with the digital environment. Using a 3D astronomy simulation that leverages a VR headset and tablet computers, this paper builds on the ideas described in eye-gaze studies by developing and implementing a metric of shared viewing across multiple devices. Preliminary findings suggest that a higher level of shared view could be related to increased conceptual discussion, as well as point to an early-stage pattern of behavior of decreased SV to prompt facilitator intervention to refocus collaborative efforts. We hope this metric will be a promising first step in further understanding and assessing the quality of collaboration across multiple device platforms in a single shared space. This paper provides an in depth look at a highly exploratory stage of a broader research trajectory to establish a robust, effective way to track screen views, including providing resources to teachers when students engage in similar learning environments, and providing insight from log data to understand how students effectively collaborate.