We present three user studies that gradually prepare our prototype system SmartHelm for use in the field, i.e. supporting cargo cyclists on public roads for cargo delivery. SmartHelm is an attention-sensitive smart helmet that integrates none-invasive brain and eye activity detection with hands-free Augmented Reality (AR) components in a speech-enabled outdoor assistance system. The described studies systematically increased in ecological validity from lab to field. The first study consisted of an Augmented Reality preparation examination in the lab. The second study then investigated simulated attention distraction modeling, whereas the third study examined real-world attention distraction modeling while cycling in traffic. During these three studies, multimodal data (EEG, eye-tracking, video, GPS and speech) has been collected synchronously and analyzed in offline and online experiments. Machine Learning models were trained and optimized for attention modeling.Results: Analyses of self-report and objective data during the simulation study show the plausibility of the simulated internal and external distractions. The analysis of behavioral data captured by multimodal biosignals recorded in the field study further shows that real visual attention distractions can be automatically identified using synchronized video and eye-tracking data. Machine Learning methods based on long short-term memory models (LSTMs) indicate that simulated attention distractions can be automatically detected from EEG data, with the best detection performance for mental distractions. Finally, the self-report data suggest that the comfort of the SmartHelm helmet should be further improved for permanent use in road traffic.

Human Computer Interaction can be impeded by various interaction obstacles, impacting a user’s perception or cognition. In this work, we detect and discriminate such interaction obstacles from different data modalities to compensate for them through User Interface (UI) adaptation. For example, we detect memory-based obstacles from brain activity and compensate through repetition of information in the UI; we detect visual obstacles from user behavior and compensate by complementing visual with auditory information in the UI. Online cognitive adaptive systems should be able to decide the most suitable UI adaptation given inputs from several obstacles detectors. In this paper, we employ a Bayesian fusion approach upon different underlying obstacles detectors over multiple consecutive interaction sessions. Experimental results show that the model promisingly outperforms the baseline in the first interaction with an average accuracy of 72.5% and further improves drastically in subsequent interactions with additional information, with an average accuracy of 98%.

The performance of humans interacting with computers can be impaired by several obstacles. Such obstacles are called Human Computer Interaction (HCI) obstacles. In this paper, we present an approach of detecting a transient visual HCI interaction obstacle called glare effect from logged user behaviour during system use. The glare effect describes a scenario in which sunlight shines onto the display, resulting in less distinguishable colors. For the detection of this obstacle one and two dimensional convolutional neural networks (1D convnets and 2D convnets) are utilized. The 1D convnet decides based on temporal sequences while the 2D convnet uses synthetic images created with those sequences. In order to increase the available training data a cognitive user simulator is used that implements a generative optimization algorithm to simulate behavioural data. Four ensemble-based systems are implemented, one each for 5, 10, 15 and 20 game rounds. The first two are based on 1D and the other two on 2D convnets. Each system consists of multiple models voting for the final prediction. The accuracies of these systems in the order of the number of rounds are 72.5%, 82.5%, 80% and 85%.

Human Computer Interaction (HCI) performance can be impaired by several HCI obstacles. Cognitive adaptive systems should dynamically detect such obstacles and compensate them with suitable User Interface (UI) adaptation. In this paper, we discuss the detection of two main HCI obstacles: memory-based and visual obstacles. A sequential model based on Long-Short Term Memory (LSTM) is suggested for such a detection of HCI obstacles. UI adaptations for both types of obstacles are discussed and analyzed. We investigate the classification performance on data from a user study with 17 participants. Furthermore, we also investigate the influence of different adaptation mechanisms on performance and subjective assessment. Results show advantages of the proposed sequential LSTM model: on the one hand, the LSTM outperforms the baseline random guess and also a baseline static model LDA in the detection of visual obstacles with 70.6% as an average accuracy. On the other hand, the evaluation of HCI sessions impeded by obstacles but supported with different UI adaptations shows that LSTM results well match the subjective assessment as a plausible detector of behaviour changes.

A memory-based interaction obstacle is a condition which impedes human memory during Human-Computer Interaction, for example a memory-loading secondary task. In this paper, we present an approach to detect the presence of such memory-based interaction obstacles from logged user behavior during system use. For this purpose, we use a recurrent neural network which models the resulting temporal sequences. To acquire a sufficient number of training episodes, we employ a cognitive user simulation. We evaluate the approach with data from a user test and on which we outperform a non-sequential baseline by up to 42% relative.