The development of AR OST-HMDs has expanded interaction possibilities beyond traditional HCI, fostering innovation in MMI. However, the lack of research specifically investigating the combined effects of different modalities, encompassing both input and output in AR OST-HMD, impedes a comprehensive understanding of this research intersection. This study bridges this gap by reviewing empirical studies on MMI in AR OST-HMDs conducted from 2012 to 2022, identifying the current state of research, and synthesizing knowledge on the implementation and evaluation of MMI in this context. The analysis of 30 empirical research yielded insights into the current usage and combinations of input and output modalities in AR OST-HMDs, its known effects, and the approaches used to study and evaluate them. We have identified that input and output are mostly examined as independent components, overlooking the potential effects of input-output combinations. We present opportunities and recommendations for future research in MMI in AR OST-HMD.

The rise of ubiquitous computing has led to the rapid development of intelligent systems such as augmented reality (AR). AR is being employed for various usage including common tasks such as web browsing, navigating and controlling other internet of things (IoT) devices. In these types of system usage, similar to mobile phones, the presentation of notifications is inevitable. However, studies with regards to the effective delivery of notification signals within the AR environment is still lacking. Thus, the present study aims to investigate the effects of different output modalities (visual, auditory and multimodal) for presenting notifications during simple search and selection tasks within the AR system. Twenty individuals participated in a user study where they were asked to perform a simple search and selection task while using the AR device. Participants were asked to confirm the notifications that were presented at random in each trial. Results of the user study showed that multimodal notification signals (both visual and auditory) yielded a higher notification recognition rate and were rated as the most appropriate modality and most preferred among the conditions tested. Also, it was found that speech input modality was used more frequently in response to notifications, regardless of the output type. Several design implications were derived from the findings of the study.

Efficient visual search and accurate target detection are crucial during flight operations. Despite the continuous effort to improve cockpit displays, visual complexity remains to be a problem that affects pilots’ search and detection performance. This has become more evident with the integration of high fidelity and information loaded cockpit displays of the fifth-generation fighter aircraft. One known but often overlooked solution to mitigate the problem with visual complexity is by decluttering. This study aims to provide an objective definition of visual complexity in cockpit displays and investigate its effects on visual search time and target detection accuracy. Moreover, it also aims to compare the visual search and target detection performance across different decluttering methods. Lastly, it intends to propose threat-based criteria for decluttering information in cockpit displays. The present study is divided into three sections, each addressing one of the three main objectives of this research. In study 1, visual search time and target detection accuracy were compared across three conditions that consist of displays with varying levels of visual complexity (low, medium and high). Visual complexity is operationally defined as the combination of different features (color, shape and orientation) and its variations (e.g., color: red, green, white). In study 2, differences in visual search time and target detection accuracy across four decluttering methods (dimming, dotting, small sizing and removal) and between groups (pilot and non-pilot) were analyzed. In study 3, through AHP analysis a threat scoring equation was proposed and tested. The results of the first study suggest that increased visual complexity leads to an increase in visual search time and more target detection errors. On the other hand, findings from the latter study revealed that there is a significant difference in the visual search time and target detection accuracy between different decluttering methods. Moreover, group differences were found when the threat scoring equation was applied to the targets. The results of the present study could provide inferences on how to alleviate the adverse effects of visual complexity through the utilization of effective decluttering methods, which in turn could further improve pilots’ overall performance.