This paper introduces GenAssist, a system for generating interactive Extended Reality (XR) programs from natural language prompts. Given plain text descriptions of desired programs, our system uses Retrieval-Augmented Generation (RAG) to retrieve related documentation and example code, which is then used to prompt Large Language Models (LLMs) to generate and execute hot-pluggable XR programs in real time. To ensure that the programs are written correctly to the user’s specifications, we add a closed-loop feedback mechanism using virtual cameras in the scene that iteratively refines the system’s output, mimicking the development cycle of human developers that compile and then interactively test programs. GenAssist generates scripts that can not only place multiple primitives and 3D models in various locations in a virtual scene, but it can also animate and enable user interactions with those objects. We show that across a benchmark of 50 diverse XR program prompts, our system achieves high output accuracy and program generation quality. Furthermore, we conduct a user study with 18 participants that demonstrates GenAssist’s effectiveness and usability (NASA TLX = 36.6) for XR program generation. We compare GenAssist to prior systems and show that it is significantly faster (<10 seconds per run) and requires fewer LLM calls.

As Artificial Intelligence (AI) and eXtended Reality (XR) evolve, integrating them effectively remains a challenge. Although multimodal large language models (MLLMs) offer powerful reasoning over text and images, they lack an inherent understanding of 3D space. Additionally, XR headsets are resource-constrained and cannot run these models locally. To address this gap, we introduce XaiR, a system that integrates MLLMs with XR to enable AI-driven spatial reasoning and interaction. XaiR employs a client-server architecture in which an XR headset (client) captures spatial data, generates 2D snapshots of the 3D environment, and renders augmented reality (AR) content, while a remote server runs multiple parallel MLLMs to generate contextually aware responses. Our demo showcases an XR cognitive assistant application that guides a user through a series of instructions. Deployed on a mobile AR headset, our system dynamically interprets user actions, tracks task progress in real time, and provides textual feedback and AR-guided assistance.

This paper discusses the integration of Multimodal Large Language Models (MLLMs) with Extended Reality (XR) headsets, focusing on enhancing machine understanding of physical spaces. By combining the contextual capabilities of MLLMs with the sensory inputs from XR, there is potential for more intuitive spatial interactions. However, the integration faces challenges due to the inherent limitations of MLLMs in processing 3D inputs and their significant resource demands for XR headsets. We introduce XaiR, a platform that facilitates integrating MLLMs with XR applications. XaiR uses a split architecture that offloads complex MLLM operations to a server while handling 3D world processing on the headset. This setup manages multiple input modalities, parallel models, and links them with real-time pose data, improving AR content placement in physical scenes. We tested XaiR’s effectiveness with a “cognitive assistant” application that guides users through tasks like making coffee or assembling furniture. Results from a 15-participant study shows over 90% accuracy in task guidance and 85% accuracy in AR content anchoring. Additionally, we evaluate MLLMs against human operators for cognitive assistant tasks which provides insights into the quality of the captured data as well as the current gap in performance for cognitive assistant tasks.

This paper presents NFCapsule, a light-weight, battery-free, and ingestible biomedical sensor that can potentially enable non-invasive detection of active eosinophilic esophagitis (EoE). EoE is an allergen-induced inflammatory condition of the esophagus; its diagnosis generally involves invasive, wired, and time-consuming endoscopy. In contrast, NFCapsule aims to wirelessly detect active EoE by tracking tissue impedance through an ingestible pill that the patient swallows. Specifically, recent biomedical research has shown that active EoE induces observable changes in the electrochemical impedance of the esophagus tissue due to an increase in its intercellular spacing. We design the NFCapsule pill based on RLC resonant circuits and model the target tissue as an impedance component that changes the resonant properties of the pill circuit. Further, the NFCapsule reader identifies the resonant properties of the pill by consistently monitoring the amount of energy transferred to the pill as it goes through the esophagus, and converts this information to estimates of bio-impedance. We implement NFCapsule pill prototypes with flexible polyimide PCBs and gelatin capsules (27 mm in height and 10 mm in diameter) and evaluated NFCapsule with both ionic agarose hydrogel models and ex vivo porcine esophageal tissues (no human patients involved). We show that NFCapsule maintains high classification accuracy under various practical scenarios (e.g., blockage, bending, movement, etc.) and achieves 85% average accuracy between healthy and unhealthy tissue samples.

We present a passive and non-intrusive sensing system for monitoring hand washing activity using structural vibration sensing. Proper hand washing is one of the most effective ways to limit the spread and transmission of disease, and has been especially critical during the COVID-19 pandemic. Prior approaches include direct observation and sensing-based approaches, but are limited in non-clinical settings due to operational restrictions and privacy concerns in sensitive areas such as restrooms. Our work introduces a new sensing modality for hand washing monitoring, which measures hand washing activity-induced vibration responses of sink structures, and uses those responses to monitor the presence and duration of hand washing. Primary research challenges are that vibration responses are similar for different activities, occur on different surfaces/structures, and tend to overlap/coincide. We overcome these challenges by extracting information about signal periodicity for similar activities through cepstrum-based features, leveraging hierarchical learning to differentiate activities on different surfaces, and denoting “primary/secondary” activities based on their relative frequency and importance. We evaluate our approach using real-world hand washing data across four different sink structures/locations, and achieve an average F1-score for hand washing activities of 0.95, which represents an 8.8X and 10.2X reduction in error over two different baseline approaches.