Generative AI enables educators to create interactive learning content by describing goals in natural language. However, without programming affordances such as traceability, refinement, and debugging, teachers struggle to align simulations with learners’ needs, refine them step by step, or verify that they reflect intended learning concepts. We propose a task-level abstraction approach that structures authoring as a sequence of representations, mirroring how teachers plan lessons and providing checkpoints for specification, inspection, and refinement. We instantiate this approach in SimStep, an authoring environment that scaffolds simulation design with four abstractions, including Concept Graph, Scenario Graph, Learning Goal Graph, and UI Graph, and introduces an inverse correction process to revise hidden model assumptions without requiring code manipulation. A technical evaluation shows that these abstractions preserve fidelity across transformations, while a user study with educators demonstrates their effectiveness in authoring simulations. Our work reframes AI-assisted programming as human–AI co-authoring through structured, domain-aligned abstractions.

Boundaries such as walls, windows, and doors are ubiquitous in the physical world, yet their potential in mixed reality (MR) remains underexplored. We present Unbounded, a Research through Design inquiry into object--boundary interaction (OBI). Building on prior work, we articulate a design space aimed at providing a shared language for OBI. To demonstrate its potential, we design and implement eight examples across productivity and art exploration scenarios, showcasing how OBIs can enrich and reframe everyday interactions. We further engage with six MR experts in one-on-one feedback sessions, using the design space and examples as design probes. Their reflections broaden the conceptual scope of OBI, reveal new possibilities for how the framework may be applied, and highlight implications for future MR interaction design. https://www.zhuoyuelyu.com/unbounded

Everyday object-based interactions (EOIs) and mid-air gesture interactions (MAIs) have been widely explored, yet prior work on their integration often targets narrow use cases or specific technologies, leaving designers and developers with limited guidance that generalizes across diverse EOIs and MAIs. We introduce Objestures (“Obj” + “Gestures”)—five interaction types spanning EOIs and MAIs, forming a design space for expressive uni- and bimanual interaction. To evaluate the usefulness of Objestures, we conducted an exploratory user study (N = 12) on basic 3D tasks (rotation and scaling), which showed performance comparable to the headset’s native freehand manipulation. To understand the user experience, we conducted case studies with the same participants across three applications (Sound, Draw, and Shadow), where participants found the interactions intuitive, engaging, and expressive, and indicated interest in everyday use. We further demonstrate the potential of Objestures across diverse contexts through 30 examples, and discuss limitations and implications.

Generative Artificial Intelligence (AI) models are a compelling way to introduce K-12 students to AI education using an artistic medium, and hence have drawn attention from K-12 AI educators. Previous Creative AI curricula mainly focus on Generative Adversarial Networks (GANs) while paying less attention to Autoregressive Models, Variational Autoencoders (VAEs), or other generative models, which have since become common in the field of generative AI. VAEs' latent-space structure and interpolation ability could effectively ground the interdisciplinary learning of AI, creative arts, and philosophy. Thus, we designed a lesson to teach high school students about VAEs. We developed a web-based game and used Plato's cave, a philosophical metaphor, to introduce how VAEs work. We used a Google Colab notebook for students to re-train VAEs with their hand-written digits to consolidate their understandings. Finally, we guided the exploration of creative VAE tools such as SketchRNN and MusicVAE to draw the connection between what they learned and real-world applications. This paper describes the lesson design and shares insights from the pilot studies with 22 students. We found that our approach was effective in teaching students about a novel AI concept.

Touch interaction with physical smartphones and tablets in Virtual Reality offers interesting opportunities for cross-device input. Unfortunately, any imprecision in the alignment of the visual representation of either the hand or device can impact the precision of touch and the realism of the experience. We first study a user’s ability to rely solely on preoperative feedback to perform touch interaction in VR, where no rendering of the hand is provided. Results indicate that touch in VR is possible without a visual representation of the hand, but accuracy is influenced by how the device is held and the distance traveled to the target. We then introduce a dynamic calibration algorithm to minimize the offset between the physical hand and its virtual representation. In a second study, we show that this algorithm can increase touch accuracy by 43%, and minimize depth-based “screen penetration” or “floating touch” errors.