Although the high performance of the convolutional neural networks (CNNs) for brain-computer interface (BCI) tasks based on raw electroencephalography (EEG) signals, the explanation of the prediction result remains challenging owing to their complex structure and numerous parameters. We propose a novel framework for explaining CNNs for EEG-based BCI tasks by using the perturbation-based influence scores. The method supports the interpretation of CNN classification for EEG signals at both the example-level and the feature-level. The experiments on the BCIC III-IVa dataset demonstrate that the proposed method is effective for not only the interpretation of the predictive models, but also for the improvement of the classification accuracy.

Before fully autonomous driving technology is developed, drivers are not free from the responsibility of Take-Over Request (TOR). Even with level 5 automation, a driver still can play a role as a final decision maker for various driving and non-driving functions, even though most of the tasks will be conducted by the vehicle. In this regard, the performance of the driver’s reaction to the request from the car is crucial in autonomous driving throughout and it is highly dependent on the driver’s attention. It is important to understand the state of the driver during autonomous driving and utilize this information in the driver-vehicle interaction system to enhance safety. Accordingly, it is important to extract features from information about the driver and the surrounding environment in order to increase the accuracy of the model that predicts the driver’s state. This paper aims to examine a method of predicting an attentional status based on the driver’s emotional state and explore the possibility of applying brain-computer interaction (BCI) with consideration of the driver’s type to increase accuracy. As a result, the prediction model for attentional state with emotional information was developed, and the driver’s characteristics and types were investigated. Based on the results, a driver-vehicle interaction system was proposed for the context of the autonomous vehicle in the future. Although the prediction model for attentional status was not powerful, it can be developed by considering more critical features such as driver’s characteristics and types. In addition, it can be used as a constraint for effective interaction when a driver uses the BCI system to deliver his/her decision to a vehicle.

There is a general lack of studies that are aimed at monitoring and detecting an operator's trust calibration, even though detecting someone's adjusted trust towards automation is essential to prevent misuse and disuse of automation. The goal of this article is to propose a convolutional neural network (CNN) based framework to estimate operators’ trust levels and detect their trust calibration in automation using image features of electroencephalogram (EEG) signals preserving temporal, spectral, and spatial information. Thirteen participants performed a set of automated Air Force multiattribute task battery tasks that differed in reliability (High/Low) and credibility (High/Low) levels. The proposed framework was compared with three machine learning methods—naïve bayes, support vector machine, multilayer perceptron—in terms of accuracy, sensitivity, and specificity of trust estimation and detection of trust calibration. Results of this article showed that the proposed framework had the highest performance of both trust estimation and detection of trust calibration in automation compared to the other comparison methods. This indicates that the proposed framework using the CNN classifier with the image-based EEG features could be an applicable model for estimating multilevel trust and detecting trust calibration during human-automation interaction. Also, it can help to prevent disuse and misuse of automation by estimating operators’ trust levels and monitoring their trust calibration in automation.

Common spatial pattern (CSP)-based spatial filtering has been most popularly applied to electroencephalogram (EEG) feature extraction for motor imagery (MI) classification in brain-computer interface (BCI) application. The effectiveness of CSP is highly affected by the frequency band and time window of EEG segments. Although numerous algorithms have been designed to optimize the spectral bands of CSP, most of them selected the time window in a heuristic way. This is likely to result in a suboptimal feature extraction since the time period when the brain responses to the mental tasks occurs may not be accurately detected. In this paper, we propose a novel algorithm, namely temporally constrained sparse group spatial pattern (TSGSP), for the simultaneous optimization of filter bands and time window within CSP to further boost classification accuracy of MI EEG. Specifically, spectrum-specific signals are first derived by bandpass filtering from raw EEG data at a set of overlapping filter bands. Each of the spectrum-specific signals is further segmented into multiple subseries using sliding window approach. We then devise a joint sparse optimization of filter bands and time windows with temporal smoothness constraint to extract robust CSP features under a multitask learning framework. A linear support vector machine classifier is trained on the optimized EEG features to accurately identify the MI tasks. An experimental study is implemented on three public EEG datasets (BCI Competition III dataset IIIa, BCI Competition IV datasets IIa, and BCI Competition IV dataset IIb) to validate the effectiveness of TSGSP in comparison to several other competing methods. Superior classification performance (averaged accuracies are 88.5%, 83.3%, and 84.3% for the three datasets, respectively) based on the experimental results confirms that the proposed algorithm is a promising candidate for performance improvement of MI-based BCIs.

Motor imagery (MI) based Brain-Computer Interfaces (BCIs) controlled Functional Electrical Stimulation (FES) can help people with severe neuromuscular impairments to control their limbs by bypassing peripheral nerves and muscle pathways. However, there are still four major limitations with current MI-based BCIs for FES control: 1) They require relatively longer training and the training procedures are not clear. 2) Classification of different MI tasks within the same limb is difficult 3) MI features cannot be utilized during passive hand-motions induced by FES due to movement artifacts. 4) Few FES units are available which have real-time parameter control functionality. This study addresses these limitations by applying a hybrid BCI paradigm with a modified low-cost commercial off-the-shelf Transcutaneous Electrical Nerve Stimulation (TENS) unit. Four subjects were asked to mimic visual cues to imagine either closing or opening their dominant hand at different rates, such as fast or slow. After FES was initiated, the subjects were asked to attend to visual stimulus to elicit Steady-State Visual Evoked Potential (SSVEP) to stop FES. Results of this study showed that the modified TENS unit was able to successfully control hand motion in real-time. The classification results of different MI tasks within the same hand were promising. Furthermore, all subjects could stop the FES within 6 seconds and the average completion time was 2 seconds. The results of this study could provide insights towards future research of rehabilitation for stroke patients.

Previous research has not fully examined the effect of additional sensory feedback, particularly delivered through the haptic modality, in pointing task performance with visual distractions. This study examined the effect of haptic feedback and visual distraction on pointing task performance in a 3D virtual environment. Results indicate a strong positive effect of haptic feedback on performance in terms of task time and root mean square error of motion. Level of similarity between distractor objects and the target object significantly reduced performance, and subjective ratings indicated a sense of increased task difficulty as similarity increased. Participants produced the best performance in trials where distractor objects had a different color but the same shape as the target object and constant haptic assistive feedback was provided. Overall, this study provides insight toward the effect of object features and similarity and the effect of haptic feedback on pointing task performance.

The goal of this study was to systematically compare signal characteristics and performance of three stimulation patterns that have been used to elicit steady-state somatosensory evoked potentials (SSSEPs): no pulses, random pulses, and rhythmic pulses with a consistent pattern. These three different vibrotactile stimulation patterns were provided to the fingertips of five healthy subjects by a small solenoid-type vibrating tactor. The five subjects, who were blindfolded, were asked to selectively focus their attention to either left or right fingertip flutter sensations, according to the audible cue given. Results of this study showed that small solenoid-type haptic tactors can elicit SSSEPs near the contra lateral central brain areas. There were significant differences in the resulting signals between the three paradigms, with the rhythmic pattern showing the highest classification accuracy. Moreover, the accuracy of the rhythmic pulse pattern was significantly higher than with or without random pulse patterns for the majority of subjects. The results of this study should provide insights to future research of SSSEP based BCIs and hybrid BCIs that use SSSEPS as one type of brain signal for users who are unable to use visual BCIs.

As a powerful interaction technology, haptically enhanced virtual environments (VEs) have found many useful applications. However, few studies have examined how wayfinding of users with visual impairments is affected by VE characteristics. An empirical experiment was conducted to investigate how different environmental characteristics (number of objects inside the environment, layout of the objects and density) affect task performance (completion time, completion ratio, and travel distance), perceived task difficulty, and behavior pattern (short and long pause) of users with visual impairments when they perform a wayfinding task in a desktop-based haptically enhanced VE. The present study found that the number of objects inside the environment and layout of the objects play a significant role in determining the completion time and distance traveled. Layout type also greatly affected the user’s behavioral pattern in terms of frequency of pauses. Finally, perceived task difficulty varied with different environmental characteristics. The study results should provide insight into the future research and development of haptically enhanced VEs for people with visual impairments.

As tactual information processing of visually impaired users has not been investigated sufficiently and there are few guidelines on the development of haptic user interfaces, development of haptic assistive system can pose many challenges. Despite the breadth and variety of tools available for elicitation of user requirements, no single tool adequately provides the needed information to develop such a specialized system for a unique user population where the understanding of user behaviors is limited. This article explores the state-of-art of requirements engineering, discusses the challenges in developing a haptic assistive system, and proposes a methodology of combining a controlled observation in a naturalistic setting with scenario-based design for effective and efficient user requirements elicitation. A case study of developing a haptically enhanced, collaborative learning-by-feeling science education system for visually impaired students was conducted to show validity and effectiveness of the proposed methodology. The case study showed that the methodology has a variety of benefits including the reduced uncertainty and a better fit to the natural behaviors of the user population.

For the present article a haptically enhanced 3D virtual environment was created, and this study investigates how visually impaired users perceive and explain the virtual space when haptic is the only input modality. The study investigates what factors affect the use of reference frame when the users verbally express a haptically constructed mental map and how such preference corresponds to their haptic movement in the virtual environment. In the study, gravity was the most influential cue in determining a vertical axis of a frame. When the users were asked to explain the relationship between themselves and the target object, they had the tendency to use the frame they initially chose to use. It was also noted that totally blind users were more responsive to various frames than users with lower vision and were faster in determining a term to explain spatial relationship. Furthermore, people who preferred relative frame were more likely to keep the haptic cursor closer to their body. Limited range of exploration caused lack of understanding of the space, whereas longer exploration time made them use more frames.

Brain–computer interfaces (BCI) have potential to provide a new channel of communication and control for people with severe motor disabilities. Although many empirical studies exist, few have specifically evaluated the impact of contributing factors on user performance and perception in BCI applications, especially for users with motor disabilities. This article reports the effects of luminosity contrast and stimulus duration on user performance and usage preference in a P300-based BCI application, P300 Speller. Ten participants with neuromuscular disabilities (amyotrophic lateral sclerosis and cerebral palsy) and 10 able-bodied participants were asked to spell six 10-character phrases in the P300 Speller. The overall accuracy was 76.5% for the able-bodied participants and 26.8% for participants with motor disabilities. The results showed that luminosity contrast and stimulus duration have significant effects on user performance. In addition, participants preferred high luminosity contrast with middle or short stimulus duration. However, these effects on user performance and preference varied for participants with and without motor disabilities. The results also indicated that although most participants with motor disabilities can establish BCI control, BCI illiteracy does exist. These results of the study should provide insights into the future research of the BCI systems, especially the real-world applicability of the BCI applications as a nonmuscular communication and control system for people with severe motor disabilities.

As a dynamic interaction technology, haptic interfaces enable users to utilize the ability of touch to feel and interact with virtual objects in a simulated virtual environment as if they were real...

Various assistive technologies such as haptic technology are used to help people with visual impairments comprehend complex information. Yet there is likely to be a misconception that users with the same disability category share the same user interface needs; furthermore, the majority of the literature has been oriented toward total blindness rather than low vision, possibly leading to dissatisfaction with assistive technologies and discontinuation of its use by those with low vision. The aim of this article is to advance the understanding of the needs of those with low vision especially in relation to haptic-incorporated multimodal user interfaces. A scenario-based, participatory design approach was used to explore their needs. A total of 19 user needs were systematically documented under three categories: audition (n = 5), touch (n = 11), and vision (n = 3). This article focuses on qualitatively exploring their needs and theoretically interpreting the needs in the light of previous studies.

To investigate the acceptance of assistive technology (AT) by special education teachers, the present study developed and tested hypothesized relationships among key determinants of AT acceptance such as the facilitating condition, perceived ease of use, computer self-efficacy, result demonstrability, perceived usefulness, and behavioral intention. Results from analysis of data collected from a number of special education teachers in schools for the visually and/or auditory impaired confirmed the effects hypothesized in our conceptual model of AT acceptance. In particular, perceived usefulness was a dominant factor affecting AT usage. Facilitating condition was strongly related to perceived ease of use, whereas perceived ease of use had a significant effect on computer self-efficacy. This study also found the importance of result demonstrability factor, which had significant effects on both computer self-efficacy and perceived usefulness. This study expanded and enriched a traditional technology acceptance model by further investigating determinants associated with the acceptance of AT by special education teachers for the blind and/or the deaf. In addition, the results of the present study should provide some insights into the understanding of AT acceptance and the decisions of AT utilization, as well as its distribution and training.

The overall quality of haptic user interfaces designed to support visually impaired students' science learning through sensorial feedback was systematically studied to investigate task performance and user behavior. Fourteen 6th- to 11th-grade students with visual impairments recruited from a state-funded blind school were asked to perform three main tasks (i.e., menu selection, structure exploration, and force recognition) using haptic user interfaces and a haptic device. This study used several dependent measures that are categorized into three types of variables: (a) task performance including success rate, workload, and task completion time; (b) user behavior defined as cursor movements proportionately represented from the user's cursor positional data; and (c) user preference. Results showed that interface type has significant effects on task performance, user behavior, and user preference, with varying degree of impact to participants with severe visual impairments performing the tasks. The results of this study as well as a set of refined design guidelines and principles should provide insights to the future research of haptic user interfaces that can be used when developing haptically enhanced science learning systems for the visually impaired.

As a nonmuscular communication and control system for people with severe motor disabilities, brain–computer interface (BCI) has found several applications. Although a few empirical studies of BCI user performance do exist, little to no research has specifically evaluated the impact of contributing factors on user performance in the BCI applications. To that end, our within-subjects design compared the impact of two different types of interface (ABC interface vs. frequency-based interface) and three levels of screen size (computer monitor, global positioning system, and cell phone screen) of a P300-based BCI application, P300 Speller, on user performance (accuracy, information transfer rate, amplitude, and latency) and usage preference. Ten participants with neuromuscular disabilities such as amyotrophic lateral sclerosis and cerebral palsy and 10 nondisabled participants were asked to type six, 10-character phrases in the P300 Speller. The overall accuracy was 79.7% for the nondisabled participants and 28.7% for participants with motor disabilities. The results showed that interface type and screen size have significant effects on user performance and usage preference, with varying degree of impact to participants with and without motor disabilities. Specifically, participants typed significantly more accurately in frequency-based interface and computer monitor screen. The results of this study should provide invaluable insights to the future research of P300-based BCI applications.

A brain–computer interface (BCI), sometimes called a direct neural interface or a brain–machine interface, detects and interprets brain signals and uses the results to communicate a user's intent (...

Despite recent advances in brain-computer interface (BCI) development, system usability still remains a large oversight. The goal of this study was to investigate the usability of a P300-based BCI system, P300 Speller, by assessing how background noise and interface color contrast affect user performance and BCI usage preference. Fifteen able-bodied participants underwent a 2 (low and high interface color contrast) × 3 (low, medium, and high background noise level) within-subjects design experiment, in which participants were asked to type six 10-character phrases in the P300 Speller paradigm. The overall accuracy in the study was 80.2%. Participants showed higher accuracy, higher information transfer rate, bigger amplitude, and smaller latency in the high interface color contrast condition than in the low contrast condition. Participants had better performance in the noisy condition than in the quiet condition, but the background noise effects were not statistically significant in the present study. These results should give some insight to the real-world applicability of the current P300 Speller as a nonmuscular communication system, especially for individuals with severe neuromuscular disabilities.

We attempted to produce an environment for education. Subjects such as mathematics or sciences are usually studied on a desk in a classroom. Our research goal is to allow students to study scientific contents more viscerally than existing studying methods by haptic interaction. To construct the environment, we have to make a user-friendly haptic interface. The study is described in two parts. The first part is defining what a useful haptic interface is. In this part, we focused on the grip of a haptic interface. SPIDAR-G is a haptic interface, which is manipulated by a grip with 8 strings.. Grip size is an important parameter for usability. We have found an optimal sphere size through SPIDAR-G usability testing. The other part is defining how teachers can use the interactive system with haptic interaction as a teaching aid. In this part , we focused on the interaction between two water molecules. First, we constructed an environment to feel Van der Waals force as well as electrostatic force with haptic interaction. Then, we observe the effectiveness of the environment when used by a class of students.