We present Social Conjurer, a system and framework that enables real-time, AI-augmented creation of virtual 3D environments for multiple users. Unlike prior GenAI systems that focus on single-user workflows or static scenes, Social Conjurer allows co-located or remote users to collaboratively build worlds using natural language, sketches, and tool-based interactions. We integrate LLMs, VLMs, and a custom networked Unity architecture to support spatial reasoning, dynamic asset placement, and shared editing. A study with 12 participants suggests the system enables co-creative worldbuilding.

We present Large Language Model for Mixed Reality (LLMR), a framework for the real-time creation and modification of interactive Mixed Reality experiences using LLMs. LLMR leverages novel strategies to tackle difficult cases where ideal training data is scarce, or where the design goal requires the synthesis of internal dynamics, intuitive analysis, or advanced interactivity. Our framework relies on text interaction and the Unity game engine. By incorporating techniques for scene understanding, task planning, self-debugging, and memory management, LLMR outperforms the standard GPT-4 by 4x in average error rate. We demonstrate LLMR’s cross-platform interoperability with several example worlds, and evaluate it on a variety of creation and modification tasks to show that it can produce and edit diverse objects, tools, and scenes. Finally, we conducted a usability study (N=11) with a diverse set that revealed participants had positive experiences with the system and would use it again.

Via Research through Design (RtD), we explore the potential of headphones as a general-purpose input device for both foreground motion-gestures as well as background sensing of user activity. As a familiar wearable device, headphones offer a compelling site for head-situated interaction and sensing. Using emerging sensing modalities such as inertial motion, capacitive touch sensing, and depth cameras, our implemented prototypes explore sensing and interaction techniques that offer a range of compelling capabilities.

Any subset of reality can potentially be interpreted as a computer, so when we speak about a particular computer, we are merely speaking about a portion of reality we can understand computationally. That means that computation is only identifiable through the human experience of it. User interface is ultimately the only grounding for the abstractions of computation, in the same way that the measurement of physical phenomena provides the only legitimate basis for physics. But user interface also changes humans. As computation is perceived, the natures of self and personhood are transformed. This process, when designers are aware of it, can be understood as an emerging form of applied philosophy or even applied spirituality.

Researchers have been working with head mounted displays and virtual reality (VR) since 1965 when Ivan Sutherland published his first paper on the subject1. This work has centered on a single user within virtual space. The literature has covered applications such as telerobotics, virtual control panels, architectural simulation and scientific visualization.Several factors now militate toward broadening the range of VR applications: developments in hardware and software, as well as a growing readiness in many fields to incorporate VR. User interface constraints of VR systems need to change in response to this changing user profile. Design of environments and behaviors in virtual worlds should be simple and accessible to experts in many fields. With the advent of multi-user systems, communications will become a major application of virtual reality.VPL has recently developed a system that allows more than one user to share a virtual space. The forms and behaviors of virtual worlds are specified graphically, so that non-programmers can design them. The system, called Reality Built for Two, will be demonstrated at the Symposium on Interactive 3D Graphics.

This paper reports on the development of a hand to machine interface device that provides real-time gesture, position and orientation information. The key element is a glove and the device as a whole incorporates a collection of technologies. Analog flex sensors on the glove measure finger bending. Hand position and orientation are measured either by ultrasonics, providing five degrees of freedom, or magnetic flux sensors, which provide six degrees of freedom. Piezoceramic benders provide the wearer of the glove with tactile feedback. These sensors are mounted on the light-weight glove and connected to the driving hardware via a small cable.