Pilots operating modern cockpits often face high cognitive demands due to complex interfaces and multitasking requirements, which can lead to overload and decreased performance. This study introduces AdaptiveCoPilot, a neuroadaptive guidance system that adapts visual, auditory, and textual cues in real time based on the pilot’s cognitive workload, measured via functional Near-Infrared Spectroscopy (fNIRS). A formative study with expert pilots (N=3) identified adaptive rules for modality switching and information load adjustments during preflight tasks. These insights informed the design of AdaptiveCoPilot, which integrates cognitive state assessments, behavioral data, and adaptive strategies within a context-aware Large Language Model (LLM). The system was evaluated in a virtual reality (VR) simulated cockpit with licensed pilots (N=8), comparing its performance against baseline and random feedback conditions. The results indicate that the pilots using AdaptiveCoPilot exhibited higher rates of optimal cognitive load states on the facets of working memory and perception, along with reduced task completion times. Based on the formative study, experimental findings, qualitative interviews, we propose a set of strategies for future development of neuroadaptive pilot guidance systems and highlight the potential of neuroadaptive systems to enhance pilot performance and safety in aviation environments.

The introduction of highly automated driving systems is expected to significantly change in-vehicle interactions, creating opportunities for the design of novel use cases and interactions for occupants. In this study, we sought to identify and extract these novel use cases and determine preliminary auditory display recommendations for these novel situations. We developed and generated use cases for level 4 automated vehicles through an expert workshop (N = 17) and online focus group interviews (N = 12). Most of the use cases we generated were then tested, apart from meditation, and user opinions were collected in a driving simulator study (N = 20). Results indicated participants were interested in functions that support their experience with both driving and non-driving related interactions in highly automated vehicles. Three categories of use cases for level 4 automated vehicles were developed: driving automation use cases, immersion use cases, and in-vehicle notification use cases. For the driving simulator study, we tested three display modalities for interaction with drivers: visual alert only, non-speech with visual, and speech with visual. In terms of situation awareness (SA), the non-speech with visual display was associated with significantly better SA for the use case consisting of a planned increase in automation level than the speech-with visual display. This study will provide guidance on sonification design to advance user experiences in highly automated vehicles.

Advances in the fields of data processing and sonification have been applied to transcribe a variety of visual experiences into an auditory format. Although image sonification examples exist, the application of these principles to visual art has not been examined thoroughly. We sought to develop and evaluate a set of guidelines for the sonification of visual artworks. Through conducting expert interviews (N = 11), we created an initial sonification algorithm that accounts for art style, lightness, and color diversity to modulate the sonified output in terms of tempo and pitch. This algorithm was evaluated through user evaluations (N = 22). User study responses supported expert interview findings, the notion that sonification can be designed to match the experience of viewing an artwork, and showed interesting interaction effects among art styles, visual components, and musical parameters. We suggest the proposed guidelines can augment visitor experiences at art exhibits and provide the basis for further experimentation.

The introduction of fully automated vehicles (FAVs) will change user experiences (UX) in personal transportation. In order for FAVs to become a life enhancing technology, it is required to design vehicular applications and user interfaces based on users’ expectations. To this end, we investigated user needs and design requirements. First, we elicited design taxonomy and use cases through literature review and trend analysis. Using these materials, expert interviews (N = 9) and focus group interviews (N = 10) were conducted. Through the qualitative analysis, we obtained twelve categories of user needs and devised design requirements based on the updated design taxonomy. While some of them have been an extension of current experiences in manual driving, completely new demands have also emerged within FAVs. Our findings contribute to designing UX in FAVs by satisfying users’ expectations and key values that can guide designers.