Procedural Content Generation via Machine Learning (PCGML) has enhanced game content creation, yet challenges in controllability and limited training data persist. This study addresses these issues by distilling a constructive PCG algorithm into a controllable PCGML model. We first generate a large amount of content with a constructive algorithm and label it using a Large Language Model (LLM). We use these synthetic labels to condition two PCGML models for content-specific generation, a diffusion model and the five-dollar model. This neural network distillation process ensures that the generation aligns with the original algorithm while introducing controllability through plain text. We define this text-conditioned PCGML as a Text-to-game-Map (T2M) task, offering an alternative to prevalent text-to-image multi-modal tasks. We compare our distilled models with the baseline constructive algorithm. Our analysis of the variety, accuracy, and quality of our generation demonstrates the efficacy of distilling constructive methods into controllable text-conditioned PCGML models.

Pilots operating modern cockpits often face high cognitive demands due to complex interfaces and multitasking requirements, which can lead to overload and decreased performance. This study introduces AdaptiveCoPilot, a neuroadaptive guidance system that adapts visual, auditory, and textual cues in real time based on the pilot’s cognitive workload, measured via functional Near-Infrared Spectroscopy (fNIRS). A formative study with expert pilots (N=3) identified adaptive rules for modality switching and information load adjustments during preflight tasks. These insights informed the design of AdaptiveCoPilot, which integrates cognitive state assessments, behavioral data, and adaptive strategies within a context-aware Large Language Model (LLM). The system was evaluated in a virtual reality (VR) simulated cockpit with licensed pilots (N=8), comparing its performance against baseline and random feedback conditions. The results indicate that the pilots using AdaptiveCoPilot exhibited higher rates of optimal cognitive load states on the facets of working memory and perception, along with reduced task completion times. Based on the formative study, experimental findings, qualitative interviews, we propose a set of strategies for future development of neuroadaptive pilot guidance systems and highlight the potential of neuroadaptive systems to enhance pilot performance and safety in aviation environments.

The concept of an intelligent augmented reality (AR) assistant has significant, wide-ranging applications, with potential uses in medicine, military, and mechanics domains. Such an assistant must be able to perceive the environment and actions, reason about the environment state in relation to a given task, and seamlessly interact with the task performer. These interactions typically involve an AR headset equipped with sensors which capture video, audio, and haptic feedback. Previous works have sought to facilitate the development of intelligent AR assistants by visualizing these sensor data streams in conjunction with the assistant's perception and reasoning model outputs. However, existing visual analytics systems do not focus on user modeling or include biometric data, and are only capable of visualizing a single task session for a single performer at a time. Moreover, they typically assume a task involves linear progression from one step to the next. We propose a visual analytics system that allows users to compare performance during multiple task sessions, focusing on non-linear tasks where different step sequences can lead to success. In particular, we design visualizations for understanding user behavior through functional near-infrared spectroscopy (fNIRS) data as a proxy for perception, attention, and memory as well as corresponding motion data (acceleration, angular velocity, and gaze). We distill these insights into embedding representations that allow users to easily select groups of sessions with similar behaviors. We provide two case studies that demonstrate how to use these visualizations to gain insights about task performance using data collected during helicopter copilot training tasks. Finally, we evaluate our approach through an in-depth examination of a think-aloud experiment with five domain experts.

Accurate and prompt delivery of Emergency Medical Services (EMS) is critical in emergency incidents, e.g., man-made or natural disaster areas. However, quickly selecting the correct EMS protocol(s) (which dictate the medical procedures to be administered to patients) in complex medical scenarios, remains a key, demanding task for Emergency Medical Technicians (EMT). In this paper, we present EMSAssist, the first end-to-end mobile voice assistant at the edge for EMS. EMSAssist consists of three major components that address technical challenges present in state-of-the-art solutions: 1) For the first time, EMSAssist proposes and applies a few-sample fine-tuning technique in medical speech recognition task, that achieves a faster and more accurate speech transcription on our EMS audio dataset, when compared to Google Cloud Speech-to-Text; 2) A WordPiece tokenizer helps boosting the end-to-end EMS protocol selection accuracy by retrieving useful information from incorrect transcriptions; 3) A novel data customization framework that enables our data-driven EMSMobileBERT model to become the new state-of-the-art for EMS protocol selection. Extensive end-to-end evaluation results at the edge show EMSAssist can more accurately select EMS protocols (Top-5 accuracy above 96%) for EMTs, with end-to-end latencies of around 4.2 seconds.

We present EMSAssist, the first end-to-end mobile voice assistant for emergency medical services (EMS). EMSAssist allows Emergency Medical Technicians (EMT) to verbally describe patients' signs and symptoms and uses EMTs' voice input to recommend top-5 EMS protocols. Through this demo, we allow the attendees to evaluate EMSAssist through a pair of Google Glass and a mobile phone. Both mobile devices can collect users' voices as input and output top-5 recommended protocols. A companion youtube video of using EMSAssist on the Google Glass is provided: https://www.youtube.com/watch?v=bj7aQJKf4aE