The extraction of shift-invariant features is an essential problem in pattern recognition. We propose an extension of the max pooling function to extract the second-best maximum value outside the window in which the maximum was found, combined with distances between the activated units to obtain the maximum values. We present eight artificial datasets of two classes with shifted 1D signals. These datasets highlight different challenges for pooling functions to extract discriminant features. The proposed approach is tested on a convolutional neural network (CNN) (weights are learned before the pooling functions) and through the extreme learning machine classifier (ELM) (weights remain random before the pooling functions). The results indicate the max pooling function’s limitations in extracting shift-invariant features in some datasets. We show that the proposed approach propagating distances related to the activated units provides the best accuracy on all the datasets. The results support the conclusion that the max pooling function has limitations in some datasets. Approaches considering additional information to extract shift-invariant high-level features, and that relationships between units can be captured through distances between units or feature maps.

Virtual reality (VR) can be utilized to create video games exploring novel ways to control and interact with the environment. VR can be employed to revisit, extend, and enhance old video games to give them a new lease of life. In this paper, we describe the game Countdown VR in which players must use mental computation to reach a target number. Mental computation is used in everyday life, it is used for quick calculations and estimations. Countdown VR is a serious game designed and implemented using the Learning Mechanics-Game Mechanics framework. We provide an analysis of different approaches for determining the difficulty level of the game based on a number and target to attain and a selection of numbers combined to reach the target. We assess the user's performance and user's experience in the game, with questionnaires to quantify the related workload, usability, flow, motivation, and potential symptoms of users playing the game. The results provide key information related to the relationships between the subjective and objective evaluations of the proposed VR game by players, with a high correlation between VR sickness questionnaires, and moderate correlations between usability and motivation, flow and motivation.

Brain-computer interfaces (BCIs) that use noninvasive electroencephalography (EEG) have to deal with the non-stationarity of the EEG signal. Capturing variations over time is not an easy task as it is dependent on many factors. In this paper, we propose to create the shift by changing the task from the training session to the test session. We propose a rapid serial visual presentation task with a covariate shift, probability shift, and concept shift. We consider two classification problems: 1) the binary classification of target vs. non-target images in a rapid serial visual presentation task, and 2) the binary classification of baseline session vs. shifted session. In the former, we consider a state-of-the-art discriminant approach using inputs time × space domain while in the latter, we consider a density-based approach with covariance matrices as inputs in different frequency bands. The results highlight the substantial drop in performance that occurs when a data shift happens (AUC≈ 0.68). The results also show that the sessions can be classified with different frequency bands. The results support the conclusion that covariate and probability shift have a substantial effect on single-trial detection and that the difference between sessions can be discriminated.

Brain-computer interface (BCI) systems facilitate unique communication between humans and computers, benefiting severely disabled individuals. Despite decades of research, BCIs are not fully integrated into clinical and commercial settings. It’s crucial to assess and explain BCI performance, offering clear explanations for potential users to avoid frustration when it doesn’t work as expected. This work investigates the efficacy of different deep learning and Riemannian geometry-based classification models in the context of motor imagery (MI) based BCI using electroencephalography (EEG). We then propose an optimal transport theory-based approach using earth mover’s distance (EMD) to quantify the comparison of the feature relevance map with the domain knowledge of neuroscience. For this, we utilized explainable AI (XAI) techniques for generating feature relevance in the spatial domain to identify important channels for model outcomes. Three state-of-the-art models are implemented - 1) Riemannian geometry-based classifier, 2) EEGNet, and 3) EEG Conformer, and the observed trend in the model’s accuracy across different architectures on the dataset correlates with the proposed feature relevance metrics. The models with diverse architectures perform significantly better when trained on channels relevant to motor imagery than data-driven channel selection. This work focuses attention on the necessity for interpretability and incorporating metrics beyond accuracy, underscores the value of combining domain knowledge and quantifying model interpretations with data-driven approaches in creating reliable and robust Brain-Computer Interfaces (BCIs).

Post-training quantization (PTQ) is a technique used to optimize and reduce the memory footprint and computational requirements of machine learning models. It has been used primarily for neural networks. For Brain-Computer Interfaces (BCI) that are fully portable and usable in various situations, it is necessary to provide approaches that are lightweight for storage and computation. In this paper, we propose the evaluation of post-training quantization on state-of-the-art approaches in brain-computer interfaces and assess their impact on accuracy. We evaluate the performance of the single-trial detection of event-related potentials representing one major BCI paradigm. The area under the receiver operating characteristic curve drops from 0.861 to 0.825 with PTQ when applied on both spatial filters and the classifier, while reducing the size of the model by about x 15. The results support the conclusion that PTQ can substantially reduce the memory footprint of the models while keeping roughly the same level of accuracy.

Virtual reality (VR) technology can be used for multiple purposes, for simulating or enhancing real-life situations. VR-based interfaces can provide new operational advantages and reduce hardware requirements, i.e., weight, by offering various enhanced input modalities that otherwise could not be used in a physical cockpit. This study presents a comparative analysis of three input modalities for a virtual keyboard in a fully immersive VR environment. Each input modality has two steps: 1) pointing to the desired button, and 2) the selection of the item. We compare the hand laser pointer with trigger selection, the head laser pointer with trigger selection, and the head laser pointer with the dwell time selection (no need to use the hands). In addition, we propose new types of visual feedback: for all the input modalities, we display in the field of view of the user the last 5 letters. For the modality with the dwell time, we add a circular progress bar around the field of view of the user to provide a constant awareness of the possible button selection. The proposed virtual keyboard input modalities were assessed by 32 participants. The results support the conclusion that the hand pointer with the trigger control provides the best speed compared to other modalities.

Children with dyslexia need specific instructions for spelling and word analysis from an early age. It is important to provide appropriate tools using technology for writing aids to such children that can help them to input text, while providing multiple feedback. However, it is unclear how children with dyslexia can efficiently use a gaze-based virtual keyboard (VK). In this study, we propose to use the typing performance of a multimodal Hindi language eye-gaze-assisted learning system based on a VK to help in the reduction of tracking errors for people with writing and reading deficiencies and to detect children with dyslexia. Performance was assessed at three levels: eye tracker, eye tracker with soft switch, and touchscreen as a baseline modality using a predefined copy-typing task. The system was validated through a series of experiments with 32 children (16 dyslexic and 16 control). The results show that the workload and the usability of the system are substantially different for children with dyslexia. Children with dyslexia have a lower typing performance when using the touchscreen modality or the eye tracker only. The detection of children with dyslexia from others was assessed with seven different types of classifiers using the typing speed on different words (AUC$>$0.9). These results highlight the need to have fully inclusive VKs. This work demonstrates the superior use of a multimodal system with participants having unique neuropsychological conditions and that the proposed system can be used to detect children with dyslexia.

Fully-immersive virtual reality (VR) can be used to display 3D data so it is possible to use different vantage points and scales to observe these data. VR provides the perception of depth that is essential to appreciate distances and positions of 3D data points. In this paper, we propose a serious game in fully immersive virtual reality related to astronomy education. It allows users to learn about the positions and types of deep sky objects that are in the Messier catalog. The present work has multiple purposes: to teach the location of Messier objects in the night sky, to learn about the main types of deep sky objects, and to raise awareness of the consequences of light pollution on the night sky as some of the Messier objects can be observed with the naked eye in dark sky locations. The usability, workload, and potential VR symptoms have been assessed with undergraduate students. The results support the conclusion that VR has a key impact on the immersion of the users in the learning activities, with a laser pointer allowing to access easily all the different functionalities. The gameplay based on visual search exploits the 3D space in VR; the hints allow users with no prior knowledge of the topic to complete the tasks. With no motion and objects located at a far distance within the environment, the VR environment does not induce any VR symptoms.

Large collections of paintings have been digitized by museums (e.g. The Metropolitan Museum of Art, New-York, USA) and companies such as Google. These resources can be provided to a wide audience through virtual reality as an instructional means to fully convey the size and the magnificence of these paintings. Accessing these resources in an immersive virtual environment can be beneficial to all the students who live far away from museums. In addition, making these resources available to a wide audience answers a current need related to the closure of many museums, because of the Covid-19 pandemic. This paper provides a description of a fully immersive virtual reality museum where 1000 paintings can be accessed in two modes: individually (one painting per room), or in galleries (multiple paintings in a room) using procedural generation. More importantly, the proposed application provides a means for museums and art instructors to insert their own collections of paintings. Such an approach aims at improving the transition of high resolution images of paintings into art galleries in virtual reality. The application has been deployed on the Steam platform, is available for free, and has been evaluated positively by users, suggesting a high interest for such an application.

Virtual reality (VR) applications using consumer grade headsets have now become popular for immersive learning. VR represents a key technology for showing to students various contents the way it would be presented in its real representation. In addition to the 3 dimensional effect, it includes an immersive dimension that can transport the user into an environment that is specific to the learning content. While teaching material using VR can be useful, such an offer must satisfy the learning objectives of the instructor and the specific needs of the course and the students. In this paper, we propose a VR museum for paintings designed for instructors in Art History. A key contribution is the definition of the painting characteristics and the questionnaires associated with the paintings in the JSON open-standard file format, facilitating the modification of the application without prior programming knowledge. Such a file can be easily edited and it contains only the key information needed by the Art History instructors. The system has been developed with Unity and tested with a Valve Index. The workload and system usability is assessed by 25 participants through a NASA-TLX and System Usability evaluation. The results support the conclusion that the system has a high usability, suggesting its potential deployment in classes of Art History to better engage students.

The availability of commercial fully immersive virtual reality systems allows the proposal and development of new applications that offer novel ways to visualize and interact with multidimensional neuroimaging data. We propose a system for the visualization and interaction with Magnetic Resonance Imaging (MRI) scans in a fully immersive learning environment in virtual reality. The system extracts the different slices from a DICOM file and presents the slices in a 3D environment where the user can display and rotate the MRI scan, and select the clipping plane in all the possible orientations. The 3D environment includes two parts: 1) a cube that displays the MRI scan in 3D and 2) three panels that include the axial, sagittal, and coronal views, where it is possible to directly access a desired slice. In addition, the environment includes a representation of the brain where it is possible to access and browse directly through the slices with the controller. This application can be used both for educational purposes as an immersive learning tool, and by neuroscience researchers as a more convenient way to browse through an MRI scan to better analyze 3D data.

Brain Computer Interfaces (BCIs) typically utilize electroencephalography (EEG) to enable control of a computer through brain signals. However, EEG is susceptible to a large amount of noise, especially from muscle activity, making it difficult to use in ubiquitous computing environments where mobility and physicality are important features. In this work, we present a novel multimodal approach for classifying the P300 event related potential (ERP) component by coupling EEG signals with nonscalp electrodes (NSE) that measure ocular and muscle artifacts. We demonstrate the effectiveness of our approach on a new dataset where the P300 signal was evoked with participants on a stationary bike under three conditions of physical activity: rest, low-intensity, and high-intensity exercise. We show that intensity of physical activity impacts the performance of both our proposed model and existing state-of-the-art models. After incorporating signals from nonscalp electrodes our proposed model performs significantly better for the physical activity conditions. Our results suggest that the incorporation of additional modalities related to eye-movements and muscle activity may improve the efficacy of mobile EEG-based BCI systems, creating the potential for ubiquitous BCI.

The non-stationary nature of electroencephalography (EEG) signals makes an EEG-based brain-computer interface (BCI) a dynamic system, thus improving its performance is a challenging task. In addition, it is well-known that due to non-stationarity based covariate shifts, the input data distributions of EEG-based BCI systems change during inter- and intra-session transitions, which poses great difficulty for developments of online adaptive data-driven systems. Ensemble learning approaches have been used previously to tackle this challenge. However, passive scheme based implementation leads to poor efficiency while increasing high computational cost. This paper presents a novel integration of covariate shift estimation and unsupervised adaptive ensemble learning (CSE-UAEL) to tackle non-stationarity in motor-imagery (MI) related EEG classification. The proposed method first employs an exponentially weighted moving average model to detect the covariate shifts in the common spatial pattern features extracted from MI related brain responses. Then, a classifier ensemble was created and updated over time to account for changes in streaming input data distribution wherein new classifiers are added to the ensemble in accordance with estimated shifts. Furthermore, using two publicly available BCI-related EEG datasets, the proposed method was extensively compared with the state-of-the-art single-classifier based passive scheme, single-classifier based active scheme and ensemble based passive schemes. The experimental results show that the proposed active scheme based ensemble learning algorithm significantly enhances the BCI performance in MI classifications.

The accelerating progress and availability of low cost computers, high speed networks, and software for high performance distributed computing allow us to reconsider computationally expensive techniques in image processing and pattern recognition. We propose a two-level hierarchical [Formula: see text]-nearest neighbor classifier where the first level uses graphics processor units (GPUs) and the second level uses a high performance cluster (HPC). The system is evaluated on the problem of character recognition with nine databases (Arabic digits, Indian digits (Bangla, Devnagari, and Oriya), Bangla characters, Indonesian characters, Arabic characters, Farsi characters and digits). Contrary to many approaches that tune the model for different scripts, the proposed image classification method is unchanged throughout the evaluation on the nine databases. We show that a hierarchical combination of decisions based on two distances, using GPUs and a HPC provides state-of-the-art performances on several scripts, and provides a better accuracy than more complex systems.

The field of artificial neural networks has a long history of several decades, where the theoretical contributions have progressed with advances in terms of power and memory in present day computers. Some old methods are now rebranded or represented, taking advantage of the power of present day computers. More particularly, we consider the current trend of Random Vector Functional Link Networks, which suggests that the architecture of a system and the learning algorithm should be properly decoupled. In this paper, we evaluate the performance of multi-layers Random Vector Functional Link Network (RVFL)/ extreme machine learning (EML) on four databases of handwritten characters. Particularly, we evaluate the impact of the architecture (number of neurons per hidden layer), and the robustness of the distribution of the results across different runs. By combining the classifier outputs from different runs, we show that such a maximum combination rule provides an accuracy of 95.97% for Arabic digits, 98.03% for Bangla, 98.64% for Devnagari, and 96.30% for Oriya digits. The results confirm that increasing the size of the hidden layers has a significant impact on the accuracy, and allows to reach state-of-the-art performance; however the performance reaches a plateau after a certain size of the hidden layers.

Connectivity analysis has become an essential tool for the evaluation of functional brain dynamics. The functional connectivity between different parts of the brain, or between different sensors, is assumed to provide key information for the discrimination of brain responses. In this study, we propose an estimation of effective cortical connectivity measures in frontal and parietal areas of human brain during four different Motor Imagery (MI) tasks. Feedback based brain-computer interface (BCI) technology has been successfully implemented for recovery of stroke patients as it can enhance the neural plasticity in brain areas associated with motor execution. However, it is still challenging to obtain reliable information regarding improvement in neural functioning during rehabilitation and its neuro-physiological dynamics. Brain connectivity is a reliable biomarker associated with brain functionality. Here, we evaluate to what extent partial granger causality can provide information in form of effective neural connectivity that can differentiate motor imagery tasks. Our results on nine subjects using the EEG dataset (BCI competition 2008 dataset 2A) show distinct connectivity patterns for all four MI classes, and higher information flow in the fronto-parietal network during task phase as compared to non-task phase. The results support the conclusion that effective connectivity analysis through partial granger causality can provide key information about neural interactions specific to different MI tasks. Moreover these interactions can be utilized as reliable biomarkers for assessment of motor recovery during stroke rehabilitation.

A major issue for bringing brain-computer interface (BCI) based on electroencephalogram (EEG) recordings outside of laboratories is the non-stationarities of EEG signals. Varying statistical properties of the signals during inter- or intra-session transfers can lead to deteriorated BCI performances over time. These variations may cause the input data distribution to shift when transitioning from the training phase (calibration session) to the testing/operating phase resulting in a covariate shift. We propose to handle this issue using a novel hybrid learning method based on two classifiers, wherein the first classifier allows including new information in the training dataset, and the second classifier performs an overall classification. The proposed method is motivated by the smoothness assumption, i.e., the points that are closest to each other are more likely to share the same label, and may be added online to enrich the training dataset. The method is evaluated on two real-world datasets corresponding to motor imagery detection (BCI competition 2008 dataset 2A and 2B). The results support the conclusion that an improvement in the classification accuracy over traditional inductive learning and semi-supervised learning methods can be obtained.

People suffering from a variety of upper and lower limb disabilities due to different neuro-muscular diseases or injuries, often find it difficult to perform day-to-day activities of mobility and grasping (pick and place) objects. This paper presents the feasibility and utility of a newly developed assistive device named EMOHEX, for disabled people to perform some activities of daily living (ADL). EMOHEX is an integrated platform that combines a low cost eye-tracking device with a powered-wheelchair mounted hand-exoskeleton, which can assist disabled people in grasping objects while moving around. A dual control panel based graphical user interface is designed wherein the user's intention to select any command button is detected through eye-tracking. The dual control consists of wheelchair control panel and exoskeleton control panel, which are interchangeable by a switch button common to both the panels. The hand-exoskeleton is capable of assisting grasp, hold, and release action. Experiments conducted on 16 healthy participants revealed that performance metrics were significantly (p<;0.01) similar for the same task complexity while for different task complexities the performance metrics were significantly (p<;0.01) different across all the participants. These results showed the feasibility and stability of the system, respectively. Moreover, the information transfer rate (ITR) of eye-tracker was found satisfactory at 55.28±1.29 bits/min and 51.02±1.72 bits/min for simple and complex task, respectively. Thus, EMOHEX has the potential as a quality assistive device for disabled people.

Over the last decade, there has been a gradual increase in the number of people with mobility and speech impairments who require novel communication devices. Most of the recent works focus on the Latin script; there is a lack of appropriate assistive devices for scripts that are specific to a country. In this paper, we propose a novel multimodal Hindi language virtual keyboard based on a menu selection with eight commands providing access to spell and type 63 different Hindi language characters along with other functionalities such as the delete command for corrections. The system has been evaluated with eight able-bodied individuals who performed a specially designed typing task. The spelling task has been achieved in three different modalities using: (i) a mouse, (ii) a portable eye-tracker, and (iii) a portable eye-tracker combined with a soft-switch. The performance has been evaluated over the changes that occur with the use of each modality in terms of typing speed and information transfer rate (ITR) at both the command and letter levels for each participant. The average speed across participants with mouse only, eye-tracker only, and eye-tracker with soft-switch were 17.12 letters/min, 10.62 letters/min, and 13.50 letters/min, respectively. The ITRs at the command and letter levels were about 67.58 bits/minute and 62.67 bits/minute, respectively, with only the eye-tracker option. Based on its robustness, the proposed system has the potential to be used as a means of augmentative communication for patients suffering from mobility and speech impairment, and can contribute to substantial improvement in their quality of life.

A common assumption in traditional supervised learning is the similar probability distribution of data between the training phase and the testing/operating phase. When transitioning from the training to testing phase, a shift in the probability distribution of input data is known as a covariate shift. Covariate shifts commonly arise in a wide range of real-world systems such as electroencephalogram-based brain–computer interfaces (BCIs). In such systems, there is a necessity for continuous monitoring of the process behavior, and tracking the state of the covariate shifts to decide about initiating adaptation in a timely manner. This paper presents a covariate shift-detection and -adaptation methodology, and its application to motor imagery-based BCIs. A covariate shift-detection test based on an exponential weighted moving average model is used to detect the covariate shift in the features extracted from motor imagery-based brain responses. Following the covariate shift-detection test, the methodology initiates an adaptation by updating the classifier during the testing/operating phase. The usefulness of the proposed method is evaluated using real-world BCI datasets (i.e. BCI competition IV dataset 2A and 2B). The results show a statistically significant improvement in the classification accuracy of the BCI system over traditional learning and semi-supervised learning methods.

New portable and noninvasive eye-trackers allow the creation of robust virtual keyboards that aim to improve the life of disabled people who are unable to communicate. This paper presents a novel multimodal virtual keyboard and evaluates the performance changes that occur with the use of different modalities. The virtual keyboard is based on a menu selection with eight main commands that allow us to spell 30 different characters and correct errors with a delete button. The system has been evaluated with 18 adult participants in three conditions corresponding to three modalities: direct selection using a mouse, with the eye-tracker to point at the desired command and a switch to select it, and with only the eye-tracker for command selection. The performance of the proposed virtual keyboard was evaluated by the speed and information transfer rate (ITR) at both the command and application levels. The average speed across subjects was 18.43 letters/min with the mouse only, 15.26 letters/min with the eye-tracker and the switch, and 9.30 letters/min with only the eye-tracker. The later provided an ITR of 44.96 and 57.46 bits/min at the letter and command levels, respectively. The results show to what extent a drop of performance can occur when switching between several modalities. While the speed decreases when controlling the virtual keyboard with the eye-tracker only, the system's performance remains functioning for severely disabled people who have their gaze as one of their only means of communication.

The recognition of handwritten digits remains a difficult problem, particularly in some scripts where there exists a large variation of style across writers. This large variability is an interesting challenge for algorithms in image processing and pattern recognition. Thanks to the accelerating progress and availability of low cost computers, high speed networks, and software for high performance distributed computing, it is possible to use computational expensive technique on large databases. In this paper, we propose to investigate the impact on the accuracy of different parameters and pre-processing methods of a distance based on image distortion models. A key challenge is to reduce the processing time of the nearest neighbor classification by considering rejection rules and adaptive distances. We propose to evaluate the performance of single character recognition on three databases of Indian handwritten digits, each database corresponds to a popular Indian script: Bangla, Devnagari, and Oriya. We show that the extraction of features related to four directions allows a significant improvement of the accuracy. The proposed approach takes advantages of GPU and high performance clusters, providing state-of-the-art performances.

In many real-world applications such as image classification, labeled training examples are difficult to obtain while unlabeled examples are readily available. In this context, semi-supervised learning methods take advantage of both labeled and unlabeled examples. In this paper, a greedy graph-based semi-supervised learning (GGSL) approach is proposed for multi-class classification problems. The labels are propagated through different graphs, which are obtained with neighborhoods of different sizes. The method assumes that nearby points share the same label, by starting with a small neighborhood where a reliable decision can be obtained, and iterates with larger neighborhoods where more examples are needed to determine the label of an example. The experimental results on toy data-sets and real data-sets, such as handwritten digit recognition, demonstrate the effectiveness of the proposed approach if a well chosen distance is used. Finally, the method does not require the tuning of hyper-parameters. We show that it is possible to achieve a recognition rate of 97.16% on handwritten digits (MNIST) while considering only one labeled example per class in the training data-set.

The detection of brain responses corresponding to the presentation of a particular class of images is a challenge in Brain-Machine Interface (BMI). Brain decoding is nowadays possible thanks to advanced brain recording devices (fMRI, EEG, MEG), and the use of appropriate signal processing and machine learning techniques. Current systems based on the detection of brain responses during rapid serial visual presentation (RSVP) tasks use EEG recording. We propose to evaluate the performance of single-trial detection with signal recorded with magnetoencephalography (MEG) during an RSVP task where participants were asked to detect images containing a person. We compare several classifiers (LDA, BLDA, k-nearest neighbor, support-vector machines) with spatial filtering, and with different sets of channels (magneto-meters, gradio-meters, all the channels). The results suggest that single-trial detection can be obtained with an AUC superior to 0.95, while typical studies based on EEG recordings using the same type of tasks, the AUC is often around 0.8. The present results show that MEG can be successfully used for target detection during a difficult RSVP task.

Learning in the presence of dataset shifts in non-stationary environments is a major challenge. Dataset shifts in the form of covariate shifts commonly occur in a broad range of real-world systems such as, electroencephalogram (EEG) based brain-computer interfaces (BCIs). Under covariate shifts, the properties of the input data distribution may shift over time from training to test/operating phase. In such systems, there is a need for continuous monitoring of the process behavior and tracking the state of the shifts to decide about initiating adaptation in a timely manner. This paper presents a covariate shift-detection and adaptation methodology, and its application to motor-imagery based BCIs. An exponential weighted moving average (EWMA) model based test is used for the covariate shift-detection in the features of EEG signals. The proposed algorithm initiates the adaptation by reconfiguring the knowledge-base of the classifier. Its performance is evaluated through experiments using a real-world dataset i.e. BCI Competition IV dataset 2A. Results show that the proposed methodology effectively performs covariate-shift-detection and adaptation and it can help to realize adaptive BCI systems.

A major problem in a brain-computer interface (BCI) based on electroencephalogram (EEG) recordings is the varying statistical properties of the signals during inter- or intra-session transfers that often lead to deteriorated BCI performances. A filter bank CSP (FBCSP) algorithm typically uses all the features from all the bands to extract and select robust features. In this paper, we evaluate the performance of four methods for frequency band selection applied to binary motor imagery classification: forward-addition (FA), backward-elimination (BE), the intersection and the union of the FA and BE. These methods automatically select and learn the best discriminative sets of frequency bands, and their corresponding CSP features. The performances of the proposed methods are evaluated on binary motor imagery classification using a publicly available real-world dataset (BCI competition 2008 dataset 2A). It is found that the BE method provides the best improvement resulting in an average classification accuracy increase of the BCI system over the FBCSP algorithm, from 77.06% to 79.09%.

Non-invasive Brain-Computer Interface (BCI) has appeared as a new hope for a large population of disabled people, who were waiting for a new communication means that would translate some brain responses into actions. After several decades of research in fields such as neuroscience and machine learning, the performance remains too low due to the low signal to noise ratio of the EEG signal, and the time that has to be dedicated to the recording of the brain responses. Hybrid BCIs consider the combination of several modalities, including brain responses, for new communication systems. The creation of a Hybrid BCI requires particular care as it possesses the constraints from several modalities. We propose to investigate the performance that could be achieved in a paradigm, where gaze control is used for the selection of an item on a computer screen and motor imagery is used to enable the selected item on the screen. Based on the results obtained from gaze detection with an eye tracker, and motor imagery detection with non-invasive EEG recording, we show that the performance of a parallel Hybrid BCI is only beneficial if the accuracy of each modality reaches a particular limit, and if the number of commands from each modality is carefully chosen.

A major challenge to devising robust brain-computer interfaces (BCIs) based on electroencephalogram (EEG) data is the immanent non-stationary characteristics of EEG signals. Statistical properties of the signals may shift during inter-or-intra session transfers that often leads to deteriorated BCI performance. The shift in the input data distribution from training to testing phase is called a covariate shift. It can be caused by various reasons such as different electrode placements, varying impedances and other ongoing brain activities. We propose an algorithm to handle this issue by adapting to the covariate shifts in the EEG data using a transductive learning approach. The performance of the proposed method is evaluated on the BCI competition 2008-Graz dataset B. The results show an improvement in classification accuracy of the BCI system over a traditional learning method. The obtained results support the conclusion that covariate-shift-adaptation using transductive learning is helpful to realize adaptive BCI systems.

To be better suited for the expectations of healthy people, new brain-computer interface (BCI) paradigms should be proposed. To tackle this problem, we investigate the emerging field of cooperative BCIs, which involves several users in a single BCI system. Because combining trials over time improves performance, combining trials across subjects can significantly improve performance compared to a single user. However, cooperative BCIs can only be used in particular settings, and new paradigms must be proposed to efficiently use this approach. To show the advantages of a cooperative BCI based on steady-state visual evoked potentials (SSVEP), we evaluate and compare the performance of combining decisions across subjects, and over time. By considering a reliable brain response such as SSVEP responses, this study represents a first evolution for the combination, and the choice of the combination method for creating a cooperative SSVEP based BCI. The results suggest that six people would be enough to obtain a perfect accuracy within one second of EEG signal.

Brain-Computer Interfaces (BCIs) represent a great challenge in signal processing and machine learning, because it is difficult to extract discriminant features corresponding to particular brain responses due to the low signal-to-noise ratio of the EEG signal. Steady-state visual evoked potentials (SSVEPs) are one of the most reliable brain responses to detect in the EEG signal. Although advanced supervised machine learning techniques can improve the classification performance of SSVEP responses, obtaining robust techniques that do not rely on training a classifier is also important. We propose to analyze, compare, and combine the performance of three state-of-the-art techniques for the detection of SSVEP responses across 10 subjects and different time segments to determine if robust classification can be obtained without subject-specific rigorous analysis using a combination of one or more techniques. The methods include two approaches based on spatial filtering, and canonical correlation analysis. The results support the conclusion that the choice of the method does not depend on the time segment, and the current techniques provide equivalent performance.

Accurate detection of single-trial event-related potentials (ERPs) in the electroencephalogram (EEG) is a difficult problem that requires efficient signal processing and machine learning techniques. Supervised spatial filtering methods that enhance the discriminative information in EEG data are commonly used to improve single-trial ERP detection. We propose a convolutional neural network (CNN) with a layer dedicated to spatial filtering for the detection of ERPs and with training based on the maximization of the area under the receiver operating characteristic curve (AUC). The CNN is compared with three common classifiers: 1) Bayesian linear discriminant analysis; 2) multilayer perceptron (MLP); and 3) support vector machines. Prior to classification, the data were spatially filtered with xDAWN (for the maximization of the signal-to-signal-plus-noise ratio), common spatial pattern, or not spatially filtered. The 12 analytical techniques were tested on EEG data recorded in three rapid serial visual presentation experiments that required the observer to discriminate rare target stimuli from frequent nontarget stimuli. Classification performance discriminating targets from nontargets depended on both the spatial filtering method and the classifier. In addition, the nonlinear classifier MLP outperformed the linear methods. Finally, training based AUC maximization provided better performance than training based on the minimization of the mean square error. The results support the conclusion that the choice of the systems architecture is critical and both spatial filtering and classification must be considered together.

The recognition of handwritten characters is an almost solved problem thanks to efficient machine learning techniques. However, the evaluation and the choice of thresholds to meet a certain level of performance remains a challenge. In this paper, we compare different rejection techniques to determine if a character has been successfully detected or not. Whereas the evaluation of binary classifiers through ROC curves and cost curves has been largely exploited in the literature, multi-class problems can involve different issues like the computational cost or the choice of having an adaptive threshold for each class. The proposed methods for rejection criteria include the maximization of the distance to the optimal performance in the ROC space. We show that optimizing this criterion is a suitable approach on two databases: Lampung handwritten characters, and Arabic handwritten digits. The results support the conclusion that in database where the accuracy is high, pair wise ROC curves analysis in multi-class problems can lead to a finer evaluation of the performance and thresholds definition.

The recognition of fully multi-oriented handwritten characters is a challenging problem. Contrary to univariate signals where the shift invariance property in the Fourier transform can be used, multivariate signals like images require special care to extract rotation invariant features. Several strategies to solve such classification tasks are possible. The proposed method considers input features obtained by the Radon transform or Polar transform. A convolutional neural network is then used for extracting higher level features. This classifier includes in addition the Fast Fourier Transform for extracting shift invariant features at the neural network level. The Radon transform and the convolutional layers process the image at the pixel level while the Fourier transform and the upper layers of the neural network process rotation invariant features. The classifier is evaluated on multi-oriented handwritten digits based on the MNIST database (Arabic digits) and on the ISI database (Bangla digits). The average recognition rate for multi-oriented characters is 93.10% for the Arabic digits and 77.01% for the Bangla digits. This neural architecture highlights the interest of the radial convolutional layer for the recognition of multi-oriented shapes.

The detection of Steady-State Visual Evoked Potential (SSVEP) responses in the Electroencephalogram (EEG) is a current challenge in signal processing applied on Brain-Computer Interfaces (BCI). BCI based on SSVEP requires visual stimuli. When these stimuli are displayed on an LCD screen, the number of frequencies for flickering object on the screen is limited. We propose to extend the number of frequencies by composing different visual patterns. We evaluate the relation ship in the frequency domain between the visual stimuli and the recorded EEG signal. The signal detection across seven types of SSVEP responses is achieved by considering spatial filters based on the generalized Rayleigh quotient. The mean detection accuracy across three subjects is 89.58%.

A Brain-Computer Interface (BCI) is a specific type of human-computer interface that enables the direct communication between human and computers by analyzing brain measurements. Oddball paradigms are used in BCI to generate event-related potentials (ERPs), like the P300 wave, on targets selected by the user. A P300 speller is based on this principle, where the detection of P300 waves allows the user to write characters. The P300 speller is composed of two classification problems. The first classification is to detect the presence of a P300 in the electroencephalogram (EEG). The second one corresponds to the combination of different P300 responses for determining the right character to spell. A new method for the detection of P300 waves is presented. This model is based on a convolutional neural network (CNN). The topology of the network is adapted to the detection of P300 waves in the time domain. Seven classifiers based on the CNN are proposed: four single classifiers with different features set and three multiclassifiers. These models are tested and compared on the Data set II of the third BCI competition. The best result is obtained with a multiclassifier solution with a recognition rate of 95.5 percent, without channel selection before the classification. The proposed approach provides also a new way for analyzing brain activities due to the receptive field of the CNN models.

For a document class, one challenge in document binarization is to automatically find a set of techniques, which are adapted to the different degradation level of the images. It is important to know the methods to use and where they can be applied advantageously. A multi-classifiers solution is presented for pixel classification. These classifiers act as filters and are used for binarization. The technique starts by clustering close pixels by K-means. A classifier, which corresponds to a supervised neural network, is dedicated to each cluster. They are trained according to a binarized image where its pixels are weighted function to erosion transformation effects. The presented method is compared to classical binarization techniques in the literature. Its effect on the commercial OCR performance reaches a gain from 0.16% for Finereader7 and 1.06% for Omnipage14 for the recognition rate.

In BCI (brain - computer interface) systems, brain signals must be processed to identify distinct activities that convey different mental states. We propose a new technique for the classification of electroencephalographic (EEG) steady-state visual evoked potential (SSVEP) activity for non-invasive BCI. The proposed method is based on a convolutional neural network that includes a Fourier transform between hidden layers in order to switch from the time domain to the frequency domain analysis in the network. The first step allows the creation of different channels. The second step is dedicated to the transformation of the signal in the frequency domain. The last step is the classification. It uses a hybrid rejection strategy that uses a junk class for the mental transition states and thresholds for the confidence values. The presented results with offline processing are obtained with 6 electrodes on 2 subjects with a time segment of 1s. The system is reliable for both subjects over 95%, with rejection criterion.

In this paper, we propose a rejection strategy for convolutional neural network models. The purpose of this work is to adapt the network's topology injunction of the geometrical error. A self-organizing map is used to change the links between the layers leading to a geometric image transformation occurring directly inside the network. Instead of learning all the possible deformation of a pattern, ambiguous patterns are rejected and the network's topology is modified in function of their geometric errors thanks to a specialized self-organizing map. Our objective is to show how an adaptive topology, without a new learning, can improve the recognition of rejected patterns in the case of handwritten digits.

In spite of the improvement of commercial optical character recognition (OCR) during the last years, their ability to process different kinds of documents can also be a default. They cannot produce a perfect recognition for all documents. However they allow producing high result for standard cases. We propose in this paper a model combining several OCRs and a specialized ICR (intelligent character recognition) based on a convolutional neural network to complement them. Instead of just performing several OCRs in parallel and applying a fusing rule of the results, a specialized neural network with an adaptive topology is added to complement the OCRs in function of the OCRs errors. This system has been tested on ancient documents containing old characters and old fonts not used in contemporary documents. The OCRs combination increases the recognition of about 3% whereas the ICR improves the recognition of rejected characters of more than 5%.