Facial expressions play an important role in human communication by conveying a wide range of emotions and intentions. While recent advances in generative models have enabled the synthesis of facial expressions from various prompts-including text, images, and audio-these approaches often overlook the direct physiological basis of facial movements. In this work, we present the first framework that uses surface electromyography (sEMG) signals, recorded from facial muscles during expression tasks, to guide a denoising diffusion probabilistic model (DDPM) for facial expression image generation. Our method employs a CNN-LSTM network to extract spatial-temporal features from sEMG signals, which are then integrated with an identity frame (a neutral facial image of the target subject) to serve as a conditional prompt for the DDPM. Experimental results demonstrate that sEMG signals can effectively guide the generation of realistic and expressive facial images, achieving a Frechet Inception Distance (FID) of 96.67, Learned Perceptual Image Patch Similarity (LPIPS) of 0.08, and an expression recognition accuracy of $100 \%$. These findings highlight the potential of sEMG signals as a novel condition prompt for generative models. This work pioneers the use of sEMG signals in facial expression synthesis, offering a new paradigm for applications in human-computer interaction, virtual reality, and medical rehabilitation, where precise and personalized expression control is critical.

Sparse Bayesian learning has promoted many effective frameworks of brain activity decoding for the brain-computer interface, including the direct reconstruction of muscle activity using brain recordings. However, existing sparse Bayesian learning algorithms mainly use Gaussian distribution as error assumption in the reconstruction task, which is not necessarily the truth in the real-world application. On the other hand, brain recording is known to be highly noisy and contains many non-Gaussian noises, which could lead to large performance degradation for sparse Bayesian learning algorithms. The goal of this paper is to propose a novel robust implementation of sparse Bayesian learning so that robustness and sparseness can be realized simultaneously. Motivated by the exceptional robustness of maximum correntropy criterion (MCC), we proposed integrating MCC to the sparse Bayesian learning regime. To be specific, we derived the explicit error assumption inherent in the MCC, and then leveraged it for the likelihood function. Meanwhile, we utilized the automatic relevance determination technique as the sparse prior distribution. To fully evaluate the proposed method, a synthetic example and a real-world muscle activity reconstruction task with two different brain modalities were leveraged. Experimental results showed, our proposed sparse Bayesian correntropy learning framework significantly improves the robustness for the noisy regression tasks. Our proposed algorithm could realize higher correlation coefficients and lower root mean squared errors for the real-world muscle activity reconstruction scenario. Sparse Bayesian correntropy learning provides a powerful approach for brain activity decoding which will promote the development of brain-computer interface technology.

Bayesian learning provides a unified skeleton to solve the electrophysiological source imaging task. From this perspective, existing source imaging algorithms utilize the Gaussian assumption for the observation noise to build the likelihood function for Bayesian inference. However, the electromagnetic measurements of brain activity are usually affected by miscellaneous artifacts, leading to a potentially non-Gaussian distribution for the observation noise. Hence the conventional Gaussian likelihood model is a suboptimal choice for the real-world source imaging task. In this study, we aim to solve this problem by proposing a new likelihood model which is robust with respect to non-Gaussian noises. Motivated by the robust maximum correntropy criterion, we propose a new improper distribution model concerning the noise assumption. This new noise distribution is leveraged to structure a robust likelihood function and integrated with hierarchical prior distributions to estimate source activities by variational inference. In particular, the score matching is adopted to determine the hyperparameters for the improper likelihood model. A comprehensive performance evaluation is performed to compare the proposed noise assumption to the conventional Gaussian model. Simulation results show that, the proposed method can realize more precise source reconstruction by designing known ground-truth. The real-world dataset also demonstrates the superiority of our new method with the visual perception task. This study provides a new backbone for Bayesian source imaging, which would facilitate its application using real-world noisy brain signal.

An essential aspect of human motor learning is the formation of inverse models, which map desired actions to motor commands. Inverse models can be learned by adjusting parameters in neural circuits to minimize errors in the performance of motor tasks through gradient descent. However, the theory of gradient descent establishes limits on the learning speed. Specifically, the eigenvalues of the Hessian of the error surface around a minimum determine the maximum speed of learning in a task. Here, we use this theoretical framework to analyze the speed of learning in different inverse model learning architectures in a set of isometric arm-reaching tasks. We show theoretically that, in these tasks, the error surface and, thus the speed of learning, are determined by the shapes of the force manipulability ellipsoid of the arm and the distribution of targets in the task. In particular, rounder manipulability ellipsoids generate a rounder error surface, allowing for faster learning of the inverse model. Rounder target distributions have a similar effect. We tested these predictions experimentally in a quasi-isometric reaching task with a visuomotor transformation. The experimental results were consistent with our theoretical predictions. Furthermore, our analysis accounts for the speed of learning in previous experiments with incompatible and compatible virtual surgery tasks, and with visuomotor rotation tasks with different numbers of targets. By identifying aspects of a task that influence the speed of learning, our results provide theoretical principles for the design of motor tasks that allow for faster learning.

Sparseness and robustness are two important properties for many machine learning scenarios. In the present study, regarding the maximum correntropy criterion (MCC) based robust regression algorithm, we investigate to integrate the MCC method with the automatic relevance determination (ARD) technique in a Bayesian framework, so that MCC-based robust regression could be implemented with ‘adaptive sparseness’. To be specific, we use an inherent noise assumption from the MCC to derive an explicit likelihood function, and realize the maximum a posteriori (MAP) estimation with the ARD prior by variational Bayesian inference. Compared to the existing robust and sparse$L_{1}$-regularized MCC regression, the proposed MCC-ARD regression can eradicate the troublesome tuning for the regularization hyper-parameter which controls the regularization strength. Further, MCC-ARD achieves superior prediction performance and feature selection capability than$L_{1}$-regularized MCC, as demonstrated by a noisy and high-dimensional simulation study. The codes of the proposed method are shared at https://sites.google.com/view/liyuanhao/codes.

The minimum error entropy (MEE) criterion is a powerful approach for non-Gaussian signal processing and robust machine learning. However, the instantiation of MEE on robust classification is a rather vacancy in the literature. The original MEE purely focuses on minimizing Renyi's quadratic entropy of the prediction errors, which could exhibit inferior capability in noisy classification tasks. To this end, we analyze the optimal error distribution with adverse outliers and introduce a specific codebook for restriction, which optimizes the error distribution toward the optimal case. Half-quadratic-based optimization and convergence analysis of the proposed learning criterion, called restricted MEE (RMEE), are provided. The experimental results considering logistic regression and extreme learning machine on synthetic data and UCI datasets, respectively, are presented to demonstrate the superior robustness of RMEE. Furthermore, we evaluate RMEE on a noisy electroencephalogram dataset, so as to strengthen its practical impact.

A significant problem in brain-computer interface (BCI) research is decoding - obtaining required information from very weak noisy electroencephalograph signals and extracting considerable information from limited data. Traditional intention decoding methods, which obtain information from induced or spontaneous brain activity, have shortcomings in terms of performance, computational expense and usage burden. Here, a new methodology called prediction error decoding was used for motor imagery (MI) detection and compared with direct intention decoding. Galvanic vestibular stimulation (GVS) was used to induce subliminal sensory feedback between the forehead and mastoids without any burden. Prediction errors were generated between the GVS-induced sensory feedback and the MI direction. The corresponding prediction error decoding of the front/back MI task was validated. A test decoding accuracy of 77.83-78.86% (median) was achieved during GVS for every 100[Formula: see text]ms interval. A nonzero weight parameter-based channel screening (WPS) method was proposed to select channels individually and commonly during GVS. When the WPS common-selected mode was compared with the WPS individual-selected mode and a classical channel selection method based on correlation coefficients (CCS), a satisfactory decoding performance of the selected channels was observed. The results indicated the positive impact of measuring common specific channels of the BCI.

Optimal feedback control is an established framework that is used to characterize human movement. However, it is not fully understood how the brain computes optimal gains through interactions with the environment. In the past study, we proposed a model of motor learning that identifies a set of feedback and feedforward controllers and a state predictor of the arm musculoskeletal system to control free reaching movements. In this study, we applied the model to force field adaptation tasks where normal reaching movements are disturbed by an external force imposed on the hand. Without a priori knowledge about the arm and environment, the model was able to adapt to the force field by generating counteracting forces to overcome it in a manner similar to what is reported in the behavioral literature. The kinematics of the movements generated by our model share characteristic features of human movements observed before and after force field adaptation. In addition, we demonstrate that the structure and learning algorithm introduced in our model induced a shift in the end-point's equilibrium position and a static force modulation, accompanied by a fast and a slow learning process. Importantly, our model does not require desired trajectories, yields movements without specifying movement duration, and predicts force generation patterns by exploring the environment. Our model demonstrates a possible mechanism through which the central nervous system may control and adapt a point-to-point reaching movement without specifying a desired trajectory by continuously updating the body's musculoskeletal model.

Electroencephalography (EEG) signals do not show electrical neuronal activity at source level. Thus, in EEG-based brain machine interface, information relatively easy to occur, such as motor imagery, has been used. Independent component analysis is a method to estimate multiple sources from EEG signals. If independent components can take some parts encoded by movement direction, it could improve classification performance for brain-machine interface. Therefore, this study aimed to find features contributing to specific direction in planning phase from independent components. Subjects performed a reaching task and were instructed to move their hand to one of five directions. Independent components were calculated from EEG signals during planning phase. We constructed feature vectors of each independent component with analysis of variance (ANOVA) and performed direction classification. Our results showed that using event-related potentials of independent components, classification accuracy were higher than chance level. Independent components that achieved high performance were associated with local region in the brain. Times and duration contributing to high performance were not identical to one another. We confirmed that independent components related to intended direction can be extracted and contributed to high accuracy.

Videogames are powerful tools to elicit emotions in players because they use different resources (visual, audio, game mechanics, story-telling, etc.) to engage the players' attention and enhance emotional experiences according to each videogame gender. Some videogames have been used as tools to elicit emotional responses from players in emotion recognition experiments, mostly related to game development scenarios. However, so far those videogames focus on elicit a particular set of emotions related to boredom, stress, fear or engagement, limiting the possibilities on using videogames to study emotion recognition, using a wider emotional spectrum. This paper presents the design and evaluation of a videogame as a tool to elicit different arousal - valence levels, and discrete emotions, for emotion recognition experiments under human-computer interaction scenarios. We performed a statistical analysis through self-assessed emotional responses from a group of 21 participants, using emotional scales and discrete emotions selection to identify the player experience in a 2D platform game. We found that with the structure proposed it is possible to elicit different emotions with the same tool, however, when we evaluated the performance across time, we found that the engagement is lost and the participants reported more answers related to boredom.

Implantable brain machine interfaces (BMI) enable severely disabled people high-performance real-time robot control and communication, utilizing high-quality intracranial neural signals. Electrocorticograms (ECoG) are useful for implantable BMIs because of not only their zero time-lag property but their high spatiotemporal resolution with long term stability also. Fully implantable devices for ECoG recording offer long-term home-use with 24/7 supports. This will help not only patients with restoring motor and communication control but also help their caregivers with reducing burdens of caregiving day and night. Until now, we established ECoG-based robot control and communication. High gamma activity (80-150 Hz) was a good decoding feature for ECoG-based real time decoding and control. Independent component analyses effectively extract neural information with dimensional reduction and contribute to improving decoding accuracy. Also, we are developing a 128-channel fully-implantable BMI device (WHERBS) for long-term home-use with 24/7 supports. We completed GLP tests and non-clinical long-term implantation. The next step is a clinical trial to confirm safety and efficacy of the implantable BMI.

Biological signals are used to control robotic exoskeletons. Our research group has developed a Brain-Machine Interface (BMI)-based Occupational Therapy Assist Suit (BOTAS), which can be controlled by both electroencephalography (EEG) and electromyography (EMG) signals. This paper presents a new method for estimating the joint angles of the arms and hands for the BOTAS from EMG signals using mathematical musculoskeletal models. In six able-bodied participants, the method succeeded in estimating the flexion/extension angles of the elbow, wrist, and metacarpophalangeal (MP) joint of the index finger. In a patient with an upper cervical spinal cord injury, the method succeeded in estimating the elbow angles. The participants could control the BOTAS empowered by the method. These results suggest that the method is potentially useful for controlling robotic exoskeletons in patients with paralyzed arms and fingers.

Two aspects of facial emotions are type and intensity level. This study is an attempt to find the relation of electrode location and latencies of EEG data with specific aspects of facial emotions. A neural network was applied to single trial EEG data for classifying the type and intensity level of facial emotions. Images of two types of facial emotions (happiness and disgust) with two intensity levels (high and low) were parametrically controlled and presented as main stimuli to evoke brain signals. Results of this study indicated that data from electrodes over the right-posterior area in the time-range window 80 to 200 ms post stimulus presentation contains P100 and N170 components strongly related to the process of recognition of facial emotions. Relative to other components, N170 was strongly related to the intensity levels of facial emotion, where it could be used for classifying low and high intensity levels of facial emotion. P100 and N170 can be used for classifying the type of facial emotions, with the accuracy when using P100 being slightly higher. Time-range window 200 to 280 ms contains N2, which can also be used for classifying the intensity level of facial emotion, but its accuracy is less than that of N170. Data from time-range window 300 to 360 ms, which contains the P300 component, was not significant for classifying type and intensity level of facial emotion relative to that of other components. This study is the first to show the possibility of using other components besides P300 for EEG-based BCIs. The results also suggest the mechanism of recognition of facial emotion, started by recognizing the type of facial emotion followed by how intense the facial emotion is expressed.

We create a framework for physical interaction with virtual human. The virtual human reacts to the input of the user in realtime. The virtual human has an ocular system model, which mimics human ocular system, to generate natural and human-like gaze motions and to create inputs for the cognitive model of the virtual human. The visual input for virtual human is generated from the dynamics simulator for the virtual world. The gaze target is chosen by vying of bottom-up and top-down attentions. The motion controller for the eyes and head is based on observations and models of human gaze motion. We create a virtual boxing game as an application of proposed framework. Proposed framework succeeds to create natural gaze reaction. In addition, by changing the balance of the bottom-up and top-down attention, looks of concentration of the virtual human can be tuned.

In this study, we achieved predictable control of a wheelchair by changing the existing mapping method of the joystick, which considers the consecutive operations of a motor of a wheelchair, to a new mapping method that corresponds to the internal model of a human being. Since the existing method uses the polar coordinate system, it is not easy at all to use this method to predict either the direction of motion or the operating order for changing the position of the wheelchair according to the requirements of an operator. In order to improve the embodiment, we divided the existing joystick mapping method into two degrees of freedom-one in the vertical axis that can control the velocity and the other, in the horizontal axis for direction control. Based on this division, we implemented a wheelchair model that can be controlled by the electromyography (EMG) signal from the neck and the arm muscles of an operator. This was achieved by mapping the divided degrees of freedom onto the degrees of freedom of the neck and arm of the operator. In this case, since the operator controls the direction of motion by the joint of his/her neck, he/she can move the wheelchair in the desired direction; thus, a more intuitive human interface is implemented.

We demonstrate a new intuitive force feedback device for advanced VR applications. Force feedback for the device is tension based and is characterized by 7 degrees of freedom (DOF); 3 DOF for translation, 3 DOF for rotation, and 1 DOF for grasp. The SPIDAR-G (Space Interface Device for Artificial Reality with Grip) will allow users to interact with virtual objects naturally by manipulating two hemispherical grips located in the center of the device frame. We show how to connect the strings between each vertex of grip and each extremity of the frame in order to achieve force feedback. In addition, methodologies will be discussed for calculating translation, orientation and grasp using the length of 8 strings connected to the motors and encoders on the frame. The SPIDAR-G exhibits smooth force feedback, minimized inertia, no backlash, scalability and safety. Such features are attributed to strategic string arrangement and control that results in stable haptic rendering. The design and control of the SPIDAR-G is described in detail and the space graphic user interface system based on the proposed SPIDAR-G system is demonstrated. Experimental results validate the feasibility of the proposed device and reveal its application to virtual reality.

This paper describes the principle and experimental results of a camera system that is composed of a commercially available CCD camera, a large-aperture lens, and an intermediate screen used to measure the distance between the camera and a far-away object by using depth from focus method.

We propose a novel two-handed multi-finger haptic interface device. This system is another kind of string based hand haptic display system. DC motors, rotary encoders, pulleys, and strings are used to measure the position of eight fingertips of the user in real time. When any fingertip comes into contact with any virtual object, the system will control the tension of the strings pulled by the motors to display force feedback at that fingertip of the user. We briefly describe the system as well as the methods used for calculation of the fingertip positions and the generation of force feedback.

Continuous advances in computer technology are making it possible to construct virtual environments with an ever-increasing sense of visual realism. What is lacking are interfaces that allow users to manipulate virtual objects in an intuitive manner. In this paper, we present a 7 DOF tension-based haptic interface that allows users to not only grip an object but also to sense an object's width. We have developed a system to utilize the physical action of gripping to display grasp manipulation in virtual environments. We also present a method to calculate the position and display force associated with this gripping mechanism. Finally, we show the validity of our proposed haptic interface through examples.