Mental fatigue, a common consequence of cognitively demanding work, impairs concentration and well-being, posing long-term health risks. Distinct from drowsiness, mental fatigue is reliably measured with EEG, yet conventional setups remain too cumbersome for everyday use. To overcome this barrier, this study investigates whether EEG headphones can detect mental fatigue and recovery across two common digital break activities: playing a video game and browsing social media. We conducted an experiment with consecutive task sessions and an intermittent break, collecting self-report, performance, and EEG data. Our results show that EEG headphones can detect mental fatigue and recovery dynamics via relative alpha power, and differentiate recovery effects between break types. Social media proved more restorative than gaming, with effects persisting into the subsequent task. These findings establish needed working principles for using headphone-EEG in naturalistic fatigue and recovery research, providing a foundation for future studies.

Virtual reality (VR) finds various applications in productivity, entertainment, and training, often requiring substantial working memory and attentional resources. Effective task performance in VR relies on prioritizing relevant information and suppressing distractions through internal attention. However, current VR systems fail to account for the impact of working memory loads, leading to over or under-stimulation. In this work, we designed an adaptive system using Electroencephalography (EEG) correlates of external and internal attention to support working memory tasks. Participants engaged in a visual working memory N-Back task, where we adapted the visual complexity of distracting elements. Our study demonstrated that EEG frontal theta and parietal alpha frequency bands effectively adjust dynamic visual complexity. The adaptive system improved task performance and reduced perceived workload compared to a reverse adaptation. Furthermore, we trained a Linear Discriminant Analysis (LDA) model and achieved a classification accuracy of 79.4% for distinguishing internal and external attention states using EEG frequency features, demonstrating the feasibility of EEG-based models for real-time attention state detection. These results highlight the potential of EEG-based adaptive systems to balance distraction management and maintain user engagement without causing cognitive overload. • Developed a VR system using EEG to balance performance and engagement. • Improved task performance with subtle, natural adaptations to environment factors. • Achieved 79.4% accuracy classifying attention states using an LDA model.

Future VR environments envision adaptive and personalized interactions.To this aim, attention detection in VR settings would allow for diverse applications and improved usability.However, attention-aware VR systems based on EEG data suffer from long training periods, hindering generalizability and widespread adoption.This work addresses the challenge of person-independent, training-free VR BCI classifying internal and external attention in VR.We compared the performance of four classifiers on an EEG dataset (N=24) featuring internal and external attention labeled classes.With the goal of online adaptation, we tested overall accuracy, different window lengths of the data, and training split to optimize the trade-off between window length and classification accuracy.Our results show that models using a complete EEG band combination consistently achieve the highest accuracy, with Linear Discriminant Analysis particularly benefiting from full-band data.The window length impacts most models' performance with short windows.LDA achieved optimal accuracy around 6.3 seconds, SVM and NN around 6.5 and 6 seconds, respectively, and RF reached stability at 6 seconds.Lastly, increasing training data ratios improved accuracy gains consistently across models.We discuss the potential of machine learning to model EEG correlates of internal and external attention as online inputs for adaptive VR systems.

In this demo, we present Virtual Lab, a VR Showroom for Biosignals devices and experiments. In Virtual Lab, visitors can explore a digital twin of the real Biosignals Lab at the Cognitive Systems Lab at the University of Bremen, and interact with digital clones of real biosignals devices, including visualizations of their function and purpose.

Electroencephalography (EEG) related research faces a significant challenge of subject independence due to the variation in brain signals and responses among individuals. While deep learning models hold promise in addressing this challenge, their effectiveness depends on large datasets for training and generalization across participants. To overcome this limitation, we propose a solution to the above limitation by increasing the size and quality of training data for subject-independent auditory attention decoding (AAD) using EEG with deep learning. Specifically, our method employs a Wasserstein Generative Adversarial Network (WGAN) to generate synthetic data, with Pearson correlation filtering the most realistic samples. We evaluated this method on a publicly available dataset of selective auditory attention experiments and showed superior performance in subject-independent AAD performance. The mixed training set, consisting of both real and artificial data generated by the WGAN+Pearson Correlation Coefficient, demonstrated approximately 4% improvement in AAD accuracy for a 1-second window. These results demonstrate that deep learning remains a viable approach to overcoming data scarcity in subject-independent AAD tasks based on EEG. Moreover, the proposed method has the potential to improve the generalization and reliability of EEG classification tasks.

For Brain-Computer Interfaces to be affordable and efficient, it is worth analyzing the importance of individual EEG channels and finding the smallest subset that still provides adequate results. In this work, we applied five different feature importance approaches to three different datasets focused on internally and externally directed attention. The methods used were: Random Forest Importance, Mutual Information, Permutation Importance, Shapley Additive Explanations, and an Ablation Study. We determined the importance of EEG channels, compared the results among the algorithms through correlation analysis, and evaluated the classification performance using different subsets of channels to validate the importance rankings. The results indicate that, in line with the existing literature, electrodes located on the right parietal cortex with the alpha frequency band appear to be the most important, followed by several channels covering the left parietal and frontal lobes. For the first dataset, we were able to reduce the number of channels from 32 to 2 while increasing the accuracy from approximately 60% to 62%. In the second dataset, we reduced the number of channels from 12 to 1 while improving the accuracy from approximately 59.5% to 64%. In the third dataset, we reduced the number of channels from 19 to 1 while only slightly decreasing the accuracy from approximately 60% to 58.5%. The results of the individual feature importance methods generally exhibit positive correlations with each other. Furthermore, we demonstrated that the rankings of channels between subjects are largely positively correlated, suggesting the presence of shared patterns of neural activity across subjects. The substantial reduction in EEG channels, identification of crucial brain regions and application of channel feature importance methods to internal/external attention data, collectively advance the development of cost-effective and efficient Brain-Computer Interfaces, paving the way for future advancements in the field.

The detection of an individual's attentional state via a Brain-Computer Interface (BCI) holds significant promise, offering a multitude of possibilities, including enhancing the usability of applications and enabling timely alerts in hazardous situations. However, the variability of EEG data between individuals and the dynamic nature of recordings pose practical challenges for achieving reliable results with BCIs. Thus, conventional methods often require the collection of each person's training data prior to usage, which is then used to train an individual model for detection. Such training data collection, makes it difficult to achieve practical use. To overcome this challenge, we propose a self-improving online learning system that personalizes a person-independent model for detecting attentional state in real-time during runtime. This eliminates the need for collecting individual training data prior to usage and instead generates the necessary labels for adaptation using automatically detected error-related potentials. The system was developed based on pre-trained models of two classifiers and used to evaluate different strategies of adaptation and label generation. A statistically significant accuracy improvement of 0.088 was achieved across all available subjects, based on simulations with pre-recorded data. These results suggest that person-dependent models for attentional state detection could in the future be substituted by self-improving classifiers that do not require a dedicated training data collection.

The ability to automatically infer relevant aspects of human users’ thoughts and feelings is crucial for technologies to adapt their behaviors in complex interactions intelligently (e.g., social robots or tutoring systems). Research on multimodal analysis has demonstrated the potential of technology to provide such estimates for a broad range of internal states and processes. However, constructing robust enough approaches for deployment in real-world applications remains an open problem. The MSECP-Wild workshop series serves as a multidisciplinary forum to present and discuss research addressing this challenge. This 4th iteration focuses on addressing varying contextual conditions (e.g., throughout an interaction or across different situations and environments) in intelligent systems as a crucial barrier for more valid real-world predictions and actions. Submissions to the workshop span efforts relevant to multimodal data collection and context-sensitive modeling. These works provide important impulses for discussions of the state-of-the-art and opportunities for future research on these subjects.

Attentional shifts can occur voluntarily (endogenous control) or reflexively (exogenous control). Previous studies have shown that the neural mechanisms underlying these shifts produce different activity patterns in the brain. Changes in visual-spatial attention are usually accompanied by eye movements and a fixation on the new center of attention. In this study, we analyze the fixation-related potentials in electroencephalographic recordings of 10 participants during computer screen-based viewing tasks. During task performance, we presented salient visual distractors to evoke reflexive attention shifts. Surrounding each fixation, 0.7-second data windows were extracted and labeled as “endogenous” or “exogenous”. Averaged over all participants, the balanced classification accuracy using a person-dependent Linear Discriminant Analysis reached 59.84%. In a leave-one-participant-out approach, the average classification accuracy reached 58.48%. Differentiating attention shifts, based on fixation-related potentials, could be used to deepen the understanding of human viewing behavior or as a Brain-Computer Interface for attention-aware user interface adaptations.

The collection of electroencephalogram (EEG)data is costly, thus it is useful to explore the generation of artificial EEG signals. However, it is difficult to evaluate the quality of these signals. In this work, we generate artificial EEG signals and compare them with real EEG measurements. We analyze the signals in the time and frequency domain and conduct a survey addressed to experts in the field of EEG signal analysis. Finally, we assess whether augmenting EEG data with generated data improves the classification performance. The artificial EEG signals were generated by a progressive Wasserstein Generative Adversarial Network (Wasserstein GAN) with gradient penalty, that was trained on attention recognition data. To investigate the effect of data augmentation, the Shallow Filterbank Common Spatial Pattern network (shallow FBCSPNet) was chosen. The analysis shows that the simulated EEG signals appear realistic in both time and frequency domain. The survey found no significant differences in EEG typical features between real and simulated, but the estimation of noise level tended to be higher for the simulated signals. The data augmentation for the classification resulted in a moderate improvement in accuracy and F-score. Overall, the results show that the quality of the simulated EEG signals is comparable to the quality of the real EEG signals.

Action anticipation is an important capability for systems interacting with humans in their everyday environments, such as robots or digital assistants. Action anticipation already works remarkably well for short periods of time; however, it is still an unsolved challenge for larger time gaps. In this paper, we propose a semantic representation of previous actions for the prediction of a distribution across possible future actions. We show that this approach is able to beat a baseline prediction for as much as 5 minutes into the future.

We present three user studies that gradually prepare our prototype system SmartHelm for use in the field, i.e. supporting cargo cyclists on public roads for cargo delivery. SmartHelm is an attention-sensitive smart helmet that integrates none-invasive brain and eye activity detection with hands-free Augmented Reality (AR) components in a speech-enabled outdoor assistance system. The described studies systematically increased in ecological validity from lab to field. The first study consisted of an Augmented Reality preparation examination in the lab. The second study then investigated simulated attention distraction modeling, whereas the third study examined real-world attention distraction modeling while cycling in traffic. During these three studies, multimodal data (EEG, eye-tracking, video, GPS and speech) has been collected synchronously and analyzed in offline and online experiments. Machine Learning models were trained and optimized for attention modeling.Results: Analyses of self-report and objective data during the simulation study show the plausibility of the simulated internal and external distractions. The analysis of behavioral data captured by multimodal biosignals recorded in the field study further shows that real visual attention distractions can be automatically identified using synchronized video and eye-tracking data. Machine Learning methods based on long short-term memory models (LSTMs) indicate that simulated attention distractions can be automatically detected from EEG data, with the best detection performance for mental distractions. Finally, the self-report data suggest that the comfort of the SmartHelm helmet should be further improved for permanent use in road traffic.

People with Dementia (PwD) and their caregivers can greatly benefit from regular cognitive and social activations. However, these activations need to be engaging and likeable to take effect and to maintain long-term motivation and wellbeing. Taking this into account, finding appropriate items in large activation content catalogues can be a challenging task, which can even lead to unhappiness (”Paradox of Choice”). User-centered Recommender Systems (RS) can help to overcome this obstacle and support PwD and their caregivers in finding engaging and likeable activation contents. In this study, we investigate a dataset collected from PwD and their (in)formal caregivers who jointly used a tablet-based activation system over multiple sessions in an unconstrained care setting. The system applies a content-based recommendation approach based on explicit ratings provided by the PwD and collects audiovisual data during usage. First, we evaluate the real-world user interactions with the RS to gain knowledge about suitable evaluation parameters for our offline analyses. Second, we train a recognition model for engagement based on the audiovisual data and enrich our dataset with the automatically detected information about the PwD’s level of engagement. Last, we apply an offline analysis and compare the RS performance based on different inputs. We show that considering PwD’s level of engagement can help to further improve the rating-based RS in terms of users’ needs and, thus, support them in the activations.

In human-computer interaction (HCI), there has been a push towards open science, but to date, this has not happened consistently for HCI research utilizing brain signals due to unclear guidelines to support reuse and reproduction. To understand existing practices in the field, this paper examines 110 publications, exploring domains, applications, modalities, mental states and processes, and more. This analysis reveals variance in how authors report experiments, which creates challenges to understand, reproduce, and build on that research. It then describes an overarching experiment model that provides a formal structure for reporting HCI research with brain signals, including definitions, terminology, categories, and examples for each aspect. Multiple distinct reporting styles were identified through factor analysis and tied to different types of research. The paper concludes with recommendations and discusses future challenges. This creates actionable items from the abstract model and empirical observations to make HCI research with brain signals more reproducible and reusable.

Modeling with multimodal data in the wild poses similar challenges in human-computer and human-robot interaction (HCI, HRI). This workshop series thus blends HCI and HRI to jointly address a broad range of current topics in multimodal modeling aimed at designing intelligent systems in the wild. From addressing data scarcity in multimodal user state recognition to emotion prediction from EEG while listening to music, our third workshop in this series aims to further stimulate this important multidisciplinary exchange.

Human Computer Interaction can be impeded by various interaction obstacles, impacting a user’s perception or cognition. In this work, we detect and discriminate such interaction obstacles from different data modalities to compensate for them through User Interface (UI) adaptation. For example, we detect memory-based obstacles from brain activity and compensate through repetition of information in the UI; we detect visual obstacles from user behavior and compensate by complementing visual with auditory information in the UI. Online cognitive adaptive systems should be able to decide the most suitable UI adaptation given inputs from several obstacles detectors. In this paper, we employ a Bayesian fusion approach upon different underlying obstacles detectors over multiple consecutive interaction sessions. Experimental results show that the model promisingly outperforms the baseline in the first interaction with an average accuracy of 72.5% and further improves drastically in subsequent interactions with additional information, with an average accuracy of 98%.

The performance of humans interacting with computers can be impaired by several obstacles. Such obstacles are called Human Computer Interaction (HCI) obstacles. In this paper, we present an approach of detecting a transient visual HCI interaction obstacle called glare effect from logged user behaviour during system use. The glare effect describes a scenario in which sunlight shines onto the display, resulting in less distinguishable colors. For the detection of this obstacle one and two dimensional convolutional neural networks (1D convnets and 2D convnets) are utilized. The 1D convnet decides based on temporal sequences while the 2D convnet uses synthetic images created with those sequences. In order to increase the available training data a cognitive user simulator is used that implements a generative optimization algorithm to simulate behavioural data. Four ensemble-based systems are implemented, one each for 5, 10, 15 and 20 game rounds. The first two are based on 1D and the other two on 2D convnets. Each system consists of multiple models voting for the final prediction. The accuracies of these systems in the order of the number of rounds are 72.5%, 82.5%, 80% and 85%.

Steady-state visually evoked potentials (SSVEP) are a widely used paradigm for the detection of attended objects. However, their aid in the recognition of other attentional states has not yet been studied in detail. In this study (n=21), we assessed the benefits of including SSVEP stimuli as probes in a screen-based task to classify internal and external attention based on 16-channel EEG data offline. Previous studies have shown that the distinction between these two attentional states based on brain activity is possible. We compared several SSVEP-stimulus settings with a baseline where no SSVEP stimulus was present. Different flickering frequencies and stimulus placements were evaluated for the possibilities of different experimental setups. We found that the influence of the stimulus on the classification accuracy is highly dependent on the settings. The Linear Discriminant Analysis (LDA) performance increased significantly when an SSVEP-evoking stimulus with a low flickering frequency was present in the center of fixation. As well as when a Canonical Correlation Analysis (CCA)-coefficient was added as the SSVEP-specific feature to a generic band-power feature set. A simple training-free, person-independent threshold approach for internal and external attention detection resulted in accuracies significantly higher than chance based on SSVEP-features that were calculated only on three occipital electrodes. These results show that such stimuli can aid the recognition of internal and external attention. Thus, they can be used in experiments or applications for a more robust detection rate. Specifically, they could improve SSVEP-based BCI paradigms by adding another level of attention-awareness.

Questionnaires are among the most common research tools in virtual reality (VR) evaluations and user studies. However, transitioning from virtual worlds to the physical world to respond to VR experience questionnaires can potentially lead to systematic biases. Administering questionnaires in VR (inVRQs) is becoming more common in contemporary research. This is based on the intuitive notion that inVRQs may ease participation, reduce the Break in Presence (BIP) and avoid biases. In this paper, we perform a systematic investigation into the effects of interrupting the VR experience through questionnaires using physiological data as a continuous and objective measure of presence. In a user study (n=50), we evaluated question-asking procedures using a VR shooter with two different levels of immersion. The users rated their player experience with a questionnaire either inside or outside of VR. Our results indicate a reduced BIP for the employed inVRQ without affecting the self-reported player experience.

Auto-correction is a standard feature of mobile text entry. While the performance of state-of-the-art auto-correct methods is usually relatively high, any errors that occur are cumbersome to repair, interrupt the flow of text entry, and challenge the user's agency over the process. In this paper, we describe a system that aims to automatically identify and repair auto-correction errors. This system comprises a multi-modal classifier for detecting auto-correction errors from brain activity, eye gaze, and context information, as well as a strategy to repair such errors by replacing the erroneous correction or suggesting alternatives. We integrated both parts in a generic Android component and thus present a research platform for studying self-repairing end-to-end systems. To demonstrate its feasibility, we performed a user study to evaluate the classification performance and usability of our approach.

Detecting, modeling, and making sense of multimodal data from human users in the wild still poses numerous challenges. Starting from aspects of data quality and reliability of our measurement instruments, the multidisciplinary endeavor of developing intelligent adaptive systems in human-computer or human-robot interaction (HCI, HRI) requires a broad range of expertise and more integrative efforts to make such systems reliable, engaging, and user-friendly. At the same time, the spectrum of applications for machine learning and modeling of multimodal data in the wild keeps expanding. From the classroom to the robot-assisted operation theatre, our workshop aims to support a vibrant exchange about current trends and methods in the field of modeling multimodal data in the wild.

We developed an attention-sensitive system that is capable of playing the children's guessing game "I spy with my litte eye" with a human user. In this game, the user selects an object from a given scene and provides the system with a single-sentence clue about it. For each trial, the system tries to guess the target object. Our approach combines top-down and bottom-up machine learning for object and color detection, automatic speech recognition, natural language processing, a semantic database, eye tracking, and augmented reality. Our evaluation demonstrates performance significantly above chance level, and results for most of the individual machine learning components are encouraging. Participants reported very high levels of satisfaction and curiosity about the system. The collected data shows that our guessing game generates a complex and rich data set. We discuss the capabilities and challenges of our system and its components with respect to multimodal attention sensing.

Roughly 50 million people worldwide are currently suffering from dementia. This number is expected to triple by 2050. Dementia is characterized by a loss of cognitive function and changes in behaviour. This includes memory, language skills, and the ability to focus and pay attention. However, it has been shown that secondary therapy such as the physical, social and cognitive activation of People with Dementia (PwD) has significant positive effects. Activation impacts cognitive functioning and can help prevent the magnification of apathy, boredom, depression, and loneliness associated with dementia. Furthermore, activation can lead to higher perceived quality of life. We follow Cohen's argument that activation stimuli have to produce engagement to take effect and adopt his definition of engagement as "the act of being occupied or involved with an external stimulus".

The Everyday Activities Science and Engineering (EASE) Collaborative Research Consortium's mission to enhance the performance of cognition-enabled robots establishes its foundation in the EASE Human Activities Data Analysis Pipeline. Through collection of diverse human activity information resources, enrichment with contextually relevant annotations, and subsequent multimodal analysis of the combined data sources, the pipeline described will provide a rich resource for robot planning researchers, through incorporation in the OpenEASE cloud platform.

Mobile users rely on typing assistant mechanisms such as prediction and autocorrect. Previous studies on mobile keyboards showed decreased performance for heavy use of word prediction, which identifies a need for more research to better understand the effectiveness of predictive features for different users. Our work aims at such a better understanding of user interaction with autocorrections and the prediction panel while entering text, in particular when these approaches fail. We present a crowd-sourced mobile text entry study with 170 participants. Our mobile web application simulates autocorrection and word prediction to capture user behaviours around these features. We found that using word prediction saves an average of 3.43 characters per phrase but also adds an average of two seconds compared to actually typing the word, resulting in a negative effect on text entry speed. We also identified that the time to fix wrong autocorrections is on average 5.5 seconds but that autocorrection does not have a significant effect on typing speed.

Dealing with fear of falling is a challenge in sport climbing. Virtual reality (VR) research suggests that using physical and reality-based interaction increases the presence in VR. In this paper, we present a study that investigates the influence of physical props on presence, stress and anxiety in a VR climbing environment involving whole body movement. To help climbers overcoming fear of falling, we compared three different conditions: Climbing in reality at 10 m height, physical climbing in VR (with props attached to the climbing wall) and virtual climbing in VR using game controllers. From subjective reports and biosignals, our results show that climbing with props in VR increases the anxiety and sense of realism in VR for sport climbing. This suggests that VR in combination with physical props are an effective simulation setup to induce the sense of height.

We investigate the integration of eye-tracking and a Brain-Computer Interface into an Augmented Reality system to control a smart home environment. Through a head-mounted display, we present context-dependent control elements which the user selects by directing attention towards them. We show that the combination of both modalities leads to the most robust detection of selections and an interface which is accepted by its users.

Human Computer Interaction (HCI) performance can be impaired by several HCI obstacles. Cognitive adaptive systems should dynamically detect such obstacles and compensate them with suitable User Interface (UI) adaptation. In this paper, we discuss the detection of two main HCI obstacles: memory-based and visual obstacles. A sequential model based on Long-Short Term Memory (LSTM) is suggested for such a detection of HCI obstacles. UI adaptations for both types of obstacles are discussed and analyzed. We investigate the classification performance on data from a user study with 17 participants. Furthermore, we also investigate the influence of different adaptation mechanisms on performance and subjective assessment. Results show advantages of the proposed sequential LSTM model: on the one hand, the LSTM outperforms the baseline random guess and also a baseline static model LDA in the detection of visual obstacles with 70.6% as an average accuracy. On the other hand, the evaluation of HCI sessions impeded by obstacles but supported with different UI adaptations shows that LSTM results well match the subjective assessment as a plausible detector of behaviour changes.

Multimodal signals allow us to gain insights into internal cognitive processes of a person, for example: speech and gesture analysis yields cues about hesitations, knowledgeability, or alertness, eye tracking yields information about a person's focus of attention, task, or cognitive state, EEG yields information about a person's cognitive load or information appraisal. Capturing cognitive processes is an important research tool to understand human behavior as well as a crucial part of a user model to an adaptive interactive system such as a robot or a tutoring system. As cognitive processes are often multifaceted, a comprehensive model requires the combination of multiple complementary signals. In this workshop at the ACM International Conference on Multimodal Interfaces (ICMI) conference in Boulder, Colorado, USA, we discussed the state-of-the-art in monitoring and modeling cognitive processes from multi-modal signals.

In this paper, we extract features of head pose, eye gaze, and facial expressions from video to estimate individual learners' attentional states in a classroom setting. We concentrate on the analysis of different definitions for a student's attention and show that available generic video processing components and a single video camera are sufficient to estimate the attentional state.

A memory-based interaction obstacle is a condition which impedes human memory during Human-Computer Interaction, for example a memory-loading secondary task. In this paper, we present an approach to detect the presence of such memory-based interaction obstacles from logged user behavior during system use. For this purpose, we use a recurrent neural network which models the resulting temporal sequences. To acquire a sufficient number of training episodes, we employ a cognitive user simulation. We evaluate the approach with data from a user test and on which we outperform a non-sequential baseline by up to 42% relative.

State-of-the-art auto-correction methods for predictive text entry systems work reasonably well, but can never be perfect due to the properties of human language. We present an approach for the automatic detection of erroneous auto-corrections based on brain activity and text-entry-based context features. We describe an experiment and a new system for the classification of human reactions to auto-correction errors. We show how auto-correction errors can be detected with an average accuracy of 85%.

In this paper, we show how recordings of gaze movements (via eye tracking) and brain activity (via electroencephalography) can be combined to provide an interface for implicit selection in a graphical user interface. This implicit selection works completely without manual intervention by the user. In our approach, we formulate implicit selection as a classification problem, describe the employed features and classification setup and introduce our experimental setup for collecting evaluation data. With a fully online-capable setup, we can achieve an F_0.2-score of up to 0.74 for temporal localization and a spatial localization accuracy of more than 0.95.

Any user interface which automatically interprets the user's input using natural modalities like gestures makes mistakes. System behavior depending on such mistakes will confuse the user and lead to an erroneous interaction flow. The automatic detection of error potentials in electroencephalographic data recorded from a user allows the system to detect such states of confusion and automatically bring the interaction back on track. In this work, we describe the design of such a self-correcting gesture interface, implement different strategies to deal with detected errors, use a simulation approach to analyze performance and costs of those strategies and execute a user study to evaluate user satisfaction. We show that self-correction significantly improves gesture recognition accuracy at lower costs and with higher acceptance than manual correction.

In this paper we show and evaluate the design of a novel telemanipulation system that maps proximity values, acquired inside of a gripper, to forces a user can feel through a haptic input device. The command console is complemented by input-devices that give the user an intuitive control over parameters relevant to the system. Furthermore, proximity sensors enable the autonomous alignment/centering of the gripper to objects in user-selected DoFs with the potential of aiding the user and lowering the workload. We evaluate our approach in a user study that shows that the telemanipulation system benefits from the supplementary proximity information and that the workload can indeed be reduced when the system operates with partial autonomy.

In this paper, we analyze game protocols for a Memo game for elderly users. We show how we can use generative statistical models to automatically reveal different playing strategies. We present a quantitative and qualitative evaluation of the approach on simulated and real data. We show that we can reliably detect different strategies and that we can use those strategy profiles to uncover relevant information on the players beyond pure performance measures.

In this paper, we show how a model of human cognition based on ACT-R can be improved to accurately predict cognitive performance under different workload levels. For this purpose, we propose a novel approach which uses an EEG-based workload model to (de-)activate a dummy model which runs in parallel to the actual task model. The dummy model consumes cognitive resources to reflect the effect of workload on behavior and performance. We evaluate the approach in two user studies with different tasks and show a significant reduction of prediction error.

In this paper, we investigate how an automatic task assistant which can detect and react to a user's workload level is able to support the user in a complex, dynamic task. In a user study, we design a dispatcher scenario with low and high workload conditions and compare the effect of four support strategies with different levels of intrusiveness using objective and subjective metrics. We see that a more intrusive strategy results in higher efficiency and effectiveness, but is also less accepted by the participants. We also show that the benefit of supportive behavior depends on the user's workload level, i.e. adaptation to its changes are necessary. We describe and evaluate a Brain Computer Interface that is able to provide the necessary user state detection.

We introduce BioKIT, a new Hidden Markov Model based toolkit to preprocess, model and interpret biosignals such as speech, motion, muscle and brain activities. The focus of this toolkit is to enable researchers from various communities to pursue their experiments and integrate real-time biosignal interpretation into their applications. BioKIT boosts a flexible two-layer structure with a modular C++ core that interfaces with a Python scripting layer, to facilitate development of new applications. BioKIT employs sequence-level parallelization and memory sharing across threads. Additionally, a fully integrated error blaming component facilitates in-depth analysis. A generic terminology keeps the barrier to entry for researchers from multiple fields to a minimum. We describe our onlinecapable dynamic decoder and report on initial experiments on three different tasks. The presented speech recognition experiments employ Kaldi [1] trained deep neural networks with the results set in relation to the real time factor needed to obtain them.

Functional near infrared spectroscopy (fNIRS) is becoming more and more popular as an innovative imaging modality for brain computer interfaces. A continuous (i.e. asynchronous) affective state monitoring system using fNIRS signals would be highly relevant for numerous disciplines, including adaptive user interfaces, entertainment, biofeedback, and medical applications. However, only stimulus-locked emotion recognition systems have been proposed by now. fNRIS signals of eight subjects at eight prefrontal locations have been recorded in response to three different classes of affect induction by emotional audio-visual stimuli and a neutral class. Our system evaluates short windows of five seconds length to continuously recognize affective states. We analyze hemodynamic responses, present a careful evaluation of binary classification tasks and investigate classification accuracies over the time.

In expert video analysis, the selection of certain events in a continuous video stream is a frequently occurring operation, e.g., in surveillance applications. Due to the dynamic and rich visual input, the constantly high attention and the required hand-eye coordination for mouse interaction, this is a very demanding and exhausting task. Hence, relevant events might be missed. We propose to use eye tracking and electroencephalography (EEG) as additional input modalities for event selection. From eye tracking, we derive the spatial location of a perceived event and from patterns in the EEG signal we derive its temporal location within the video stream. This reduces the amount of the required active user input in the selection process, and thus has the potential to reduce the user's workload. In this paper, we describe the employed methods for the localization processes and introduce the developed scenario in which we investigate the feasibility of this approach. Finally, we present and discuss results on the accuracy and the speed of the method and investigate how the modalities interact.

This paper presents an online multimodal person independent workload classification system using blood volume pressure, respiration measures, electrodermal activity and electroencephalography. For each modality a classifier based on linear discriminant analysis is trained. The classification results obtained on short data frames are fused using weighted majority voting. The system was trained and evaluated on a large training corpus of 152 participants, exposed to controlled and uncontrolled scenarios for inducing workload, including a driving task conducted in a realistic driving simulator. Using person dependent feature space normalization, we achieve a classification accuracy of up to 94% for discrimination of relaxed state vs. high workload.

In this work we present an online ASR system that is able to discriminate voice commands directed to an operationable screen from irrelevant speech segments. For classification of the sound segments we explored several features that are based on prosody as well as properties generated during the decoding process. For a vocabulary of 259 words and more than 10k possible commands, our realtime Verbal Command Detector managed to detect 88.3% of the commands in our evaluation data while maintaining a low False Positive Rate (FPR) of 1.5%. On an evaluation task using an episode of Star Trek, our system was able to detect 91.2% of all commands with a FPR of 1.8% with only minor adjustments. The system is part of and used in the Smart Control Room at the Fraunhofer IOSB in Karlsruhe [1], an experimental smart environment that uses multiple input modalities for crisis response.

This work describes the development and evaluation of a recognizer for different levels of cognitive workload in the car. We collected multiple biosignal streams (skin conductance, pulse, respiration, EEG) during an experiment in a driving simulator in which the drivers performed a primary driving task and several secondary tasks of varying difficulty. From this data, an SVM based workload classifier was trained and evaluated, yielding recognition rates of up to for three levels of workload.

Abstract This paper describes the integration of a cognitive memorymodel into a spoken dialog system for an in-car tourguide ap-plication. This memory model enhances the capabilities of thesystem and of the simulated user by estimating if and whichinformation is relevant and useful in a given situation. An eval-uation study with 15 human judges is performed to demonstratethe feasibility of the described approach. The results show thatthe proposed utterance selection strategy and the memory modelsignificantly improve the human-like interaction behavior of thespoken dialog system in terms of the amount and quality ofgiven information, relevance, manner, and naturalness of thespoken interaction.Index Terms: spoken interaction, cognition, memory model,workload, utterance selection 1. Introduction Spoken dialog systems (SDS) have matured to a point wherethey find their way into many real-world applications. How-ever, their application in very dynamic scenarios remains anopen and very challenging task. In our application, we imple-ment a virtual co-driver in the car that acts as a tourguide duringa ride. While a traditional SDS already offers an eyes-free andhands-free control for in-car information applications, the par-allel driving task uses the user’s cognitive capacity so we canno longer assume to deal with a fully attentive and perfect inter-action partner as in more static environments. Additionally, wehave to deal with an ever-changing context in the dynamic en-vironment. Therefore, we need to integrate components in ourdialog systems that are able to explicitly model, predict, andcope with this imperfect user and the varying focus to ensure aseamless and successful dialog experience.Especially in interaction scenarios which are not directlytask-driven like our tourguide scenario, utterance selection isnot trivial as while we still follow a clearly defined goal of pro-viding as much interesting information as possible, but have noclear order or priority of information chunks to present. Thesame is true if we want to simulate a user for evaluation or au-tomatic strategy learning. To create coherent user behavior, weneed to establish what the simulated users currently have ontheir mind. An explicit memory model aims for a detailed rep-resentation of human memory by dynamically modeling an in-dividual strength of activation for each chunk of information.This paper describes the implementation of the model and howit is applied to utterance selection for both the system and thesimulated user. The primary goal of the utterance selection is tofind an utterance that is most relevant in the current context andof most interest to the user.

We present an approach to introduce social awareness in interactive systems. The IslEnquirer is a system which automatically builds social user models. It initializes the models by social network analysis of available offline data. These models are then verified and extended by interactive learning which is carried out by a robot initiated spoken dialog with the user.

No abstract available.