Integrating motor and cognitive training is crucial for stroke patients. However, interactive technologies for assisting such synergistic trainings are underdeveloped. This article presents the design and evaluation of CalliRehab, an AR-enhanced multisensory tool for facilitating the coordinated training of upper limb motor skills and cognitive functions through calligraphy therapy tasks. The effects of CalliRehab were examined through a within-subject experiment involving 21 stroke patients. Results indicate that, compared to the non-feedback baseline, multimodal feedback improved the task accuracy with a significant reduction in error counts (p < 0.001). Participants’ cognitive load has been remarkably lowered in the NASA task load index (p < 0.001), whereas the intrinsic motivation (p < 0.001) and the technology acceptance (p < 0.001) have been increased significantly. This study confirms that AR-enhanced multisensory training tools can effectively support integrating motor and cognitive training in stroke patients, particularly in enhancing task quality and training experiences.

Autism Spectrum Disorder (ASD) impacts social interaction, communication, and cognitive functioning. In recent years, virtual reality (VR) technology, with its advantages such as strong immersive capabilities, has been increasingly applied to ASD ergotherapy. However, existing systems often overlook human factors considerations, including users' perceptual preferences and attentional load during system interactions. This study proposes a Personalized Multisensory Adaptive Roaming System (P-MARS), which integrates large-scale models to analyze users' sensory preferences and generates personalized VR scenarios combined with visual, auditory, and olfactory stimuli, aiming to enhance ergotherapy outcomes for children with ASD. A longitudinal study employing functional near-infrared spectroscopy (fNIRS) was conducted to assess participants' neural states. The results demonstrate a significant reduction in central neural activation levels among children during the later stages of training. These findings suggest that periodic VR-based training can substantially alleviate cognitive load and anxiety in children with ASD, thereby accelerating ergotherapy progress. The study further validates the critical importance and efficacy of incorporating human factors into ergotherapy system design.

In special environments such as isolated, confined, and extreme (ICE) spaces, optimizing the human-computer interaction (HCI) experience and accurately assessing the cognitive state of users is important. In this study, portable functional near-infrared spectroscopy (fNIRS) was used to monitor the changes in users’ brain activities while they performed three cognitive tasks: Stroop, Vigilance, and Reverse. The study aimed to evaluate the effects of different tasks on users’ cognitive load and provide a basis for the design optimization of HCI systems in special environments. The results demonstrate that regional brain activation shows the greatest feature importance between resting and task states, reflecting the users’ cognitive processing, while functional connectivity networks show higher stability across tasks and play a key role in state recognition. This discovery lays the groundwork for developing intelligent systems capable of real-time sensing and optimizing interactions, guiding the optimization of HCI systems in special environments.

As human-computer interaction (HCI) technology advances, the use of augmented reality (AR) in cognitive training is becoming more prevalent. However, traditional training methods often apply a one-size-fits-all approach, failing to accommodate the varied training needs of individuals with different cognitive levels. Additionally, most HCI systems use subjective questionnaires for evaluation, which can be influenced by the subjects' emotional and mental states. To overcome these challenges, this study developed an AR-based adaptive HCI cognitive training system that dynamically adjusts task difficulty based on real-time user performance. We used multi-source data to empirically validate the effectiveness of adaptive HCI in cognitive training. Specifically, we recorded functional Near-Infrared Spectroscopy (fNIRS) data, movement data, task performance, and subjective feedback from 22 elderly participants, dividing them into two groups—low cognitive group and normal cognitive group. The results showed that the system exerted a significant influence on brain functional connectivity (FC) associated with cognition, movement, and vision. Changes in FC may highlight the benefits of adaptive HCI training strategies. Furthermore, participants with normal cognitive abilities significantly outperformed their low cognitive counterparts in task performance. In conclusion, this study designed and evaluated an AR-based adaptive HCI cognitive training system that ensures personalized training. It demonstrated the feasibility of adaptive HCI strategies in cognitive rehabilitation by incorporating physiological and behavioral data, thereby enhancing the precision of quantitative assessments for HCI systems.

In the era of Industry 5.0, the rapid development of the Internet of Things (IoT) is expected to extend its applications to broader human–computer interaction (HCI) and human–machine connectivity. With the increasing number of healthcare groups, there is an urgent need to develop embodied neuromorphic intelligent human–machine connectivity products based on these technologies. However, it remains a critical challenge to address the influencing factors of such product designs and how to quantify interaction efficacy. This study proposes a product framework combining IoT with embodied neuromorphic intelligence and conducts a user study. A cognitive rehabilitation product was developed using Leap Motion technology, with gesture recognition difficulty as a design variable, and product efficacy was quantified using a combination of brain–computer interaction and multisource interactive feedback. Fifteen elderly and fifteen young participants engaged in puppet control tasks under resting, simple, and complex conditions. The study compared brain activation levels, brain network connectivity, and eight behavioral indicators. The results demonstrated that the difficulty level significantly affects interaction efficacy. This research reveals neurological changes in the rehabilitation process of healthcare groups and opens new directions for the design and efficacy evaluation of embodied neuromorphic intelligence in HCI rehabilitation products through IoT and big data analytics, thereby advancing the development of Healthcare Industry 5.0.

OBJECTIVE: The quest for scientifically effective rehabilitation methods for stroke recovery constitutes an urgent need. However, due to the inadequacies of longitudinal studies and multimodal assessment methods, the rehabilitation mechanisms of methods such as Acupuncture Treatment (AT) and Motor Imagery (MI) remain unclear. Consequently, this study presents both AT and Acupuncture Synchronized with MI (ASMI) therapies, utilizing a combination of subjective and objective approaches to evaluate the long-term impacts of these two treatment modalities. METHODS: A longitudinal design was adopted for a duration of two weeks. Clinical improvement in patients was assessed using scale data, while Functional Near-infrared spectroscopy (fNIRS) and Electroencephalogram (EEG) data were collected to analyze changes in brain function. This study proposed the Cluster-Span Threshold for Directed Networks (CSTDN) algorithm for identifying key connections within the brain network and conducted in-depth analysis using graph theory metrics. RESULTS: Scale data indicated improvements in behavioral capabilities in both groups post-treatment. EEG and fNIRS data revealed significant variations in specific frequency bands between the two groups. CONCLUSION: This study not only validates the efficacy of AT and ASMI in stroke rehabilitation but also unveils the underlying neurobiological mechanisms through multimodal data analysis. The proposed CSTDN algorithm and graph theory analysis offer new perspectives for understanding changes in the brain network. SIGNIFICANCE: This research contributes to the optimization of future rehabilitation treatment strategies and the formulation of personalized treatment plans.

The rising incidence of stroke has created a significant global public health challenge. The immersive qualities of virtual reality (VR) technology, along with its distinct advantages, make it a promising tool for stroke rehabilitation. To address this challenge, developing VR-based upper limb rehabilitation systems has become a critical research focus. This study developed and evaluated an innovative ArmVR system that combines VR technology with rehabilitation hardware to improve recovery outcomes for stroke patients. Through comprehensive assessments, including neurofeedback, pressure feedback, and subjective feedback, the results suggest that VR technology has the potential to positively support the recovery of cognitive and motor functions. Different VR environments affect rehabilitation outcomes: forest scenarios aid emotional relaxation, while city scenarios better activate motor centers in stroke patients. The study also identified variations in responses among different user groups. Normal users showed significant changes in cognitive function, whereas stroke patients primarily experienced motor function recovery. These findings suggest that VR-integrated rehabilitation systems possess great potential, and personalized design can further enhance recovery outcomes, meet diverse patient needs, and ultimately improve quality of life.

Studies have shown that prolonged Human-Computer Interaction (HCI) fatigue can increase the risk of mental illness and lead to a higher probability of errors and accidents during operations. Virtual Reality (VR) technology can simultaneously stimulate multiple senses such as visual, auditory, and tactile, providing an immersive experience that enhances cognition and understanding. Therefore, this study collects multimodal data to develop evaluation methods for HCI fatigue and further explores the fatigue-relieving effects of VR technology by comparing it with flat video. Using a modular design, electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) data in the resting, fatigue-induced, and recovery states, eye movement data in the resting and fatigue-induced states, as well as subjective scale results after each state were collected from the participants. Preprocessing and statistical analysis are performed through data flow architecture. After fatigue induction, it was found that the degree of activation of brain areas, especially the Theta band of prefrontal cortex, occurred significantly higher, the effective connectivity in the Alpha and Theta bands occurred significantly lower, the subjects' pupil diameters decreased, the blink frequency increased, and subjective questionnaire scores increased, which verified the validity of the multimodal data for assessing HCI fatigue. Analyzing fatigue relief through subgroups, it was found that when using the natural grassland scene with soothing music, both flat video and VR had the ability to alleviate fatigue, which was manifested as a significant decrease in the Alpha band in the LPFC brain area and a decrease in the questionnaire score. Moreover, during the recovery state, it was found that compared to the video group, the VR group had significantly higher activation in the Alpha and Theta bands of the prefrontal cortex, while the video group had significantly higher effective connectivity than the VR group in the Alpha band. This study delved deeply into the multidimensional characterization of fatigue and investigated new scenarios for the use of VR, which can help to promote the use of VR and can be migrated to scenarios that require fatigue management and productivity enhancement.

With the growing needs on rehabilitation of the mild cognitive impairment (MCI) users group and the advantages of virtual reality (VR) technologies in cognitive training, the development of VR-based rehabilitation training methods has become a hot spot recently. However, the challenges in accurately measuring users’ needs and quantifying training system efficacy are still not well resolved, especially for longitudinal tracking. In this study, a VR-based cognitive training and evaluation system is designed and implemented, targeting at fulfilling the rehabilitation needs of MCI users. It evaluates the impact of longitudinal VR-based training on MCI users with a number of feedback methodologies including brain activation indicators, brain network connectivity indicators, behavioral indicators and the Montreal Cognitive Assessment (MoCA) scale scores, extracted from multi-modal data collected while training. A two-month longitudinal tracking ergonomics experiment was conducted to validate the usability of the feedback methodologies and to explore the influence of the training duration on the rehabilitation efficacy. The results showed that our proposed VR-based cognitive training and evaluation system had a positively significant impact on the rehabilitation of the MCI group. Meanwhile, the multi-source feedbacks can also help the updates and iterations of VR-based rehabilitation training systems. Finally, this study provides guidance for the selection of rehabilitation cycles and emphasizes the importance of quantitative studies with longitudinal follow-up in assessing rehabilitation efficacy.

Over time, repeated mouse-dragging manipulation may cause discomfort in the upper extremities. This study compared biomechanical parameters, mouse movement data, and the discomfort perception index between 2-min mouse dragging and moving manipulation tasks with higher or normal target objectives. We recruited 20 non-symptomatic graphic design students who frequently engage in intensive mouse-dragging manipulation. We assessed the impact of continued dragging versus non-dragging manipulation using electromyographic data, mouse fingertip pressure data, and mouse trajectory velocity data. We also performed a temporal correlation analysis between EMG, mouse velocity, and fingertip pressure to investigate the relationship between physiological data and mouse movement performance. The study revealed significant differences between the dragging and moving tasks regarding muscle activity, mouse velocity, and fingertip pressure. In particular, the percentage of muscle activation on the right side of the extensor carpi radialis longus (ECR) and the lateral head of the triceps brachii (TB) differed significantly between the two tasks, with higher muscle activation levels during the dragging task. Moreover, the average muscle activation of ECR and TB was significantly higher at high target operation levels. In addition, the study revealed that the horizontal, vertical, and mean mouse velocities were significantly higher for the dragging manipulation in the high target task than for the mouse moving manipulation. In the temporal correlation analysis, the correlation coefficients of muscle activation, fingertip pressure, and mouse mean velocity differed between the two mouse manipulation behaviors, with a higher correlation between muscle activation levels and pressure values during the dragging manipulation.

Non-pharmacological treatments have gained significant attention in the field of cognitive impairment. Among them, human–computer interaction-based (HCI) methods have emerged as a promising approach due to their broad applicability and convenience in assessing symptoms associated with this progressively debilitating condition. However, existing rehabilitation training systems for cognitive impairment lack effective assessment methods to meet the diverse rehabilitation needs of users. In this article, we surveyed existing HCI-based cognitive rehabilitation training systems and analyzed their advantages and shortcomings. Drawing from the insights gained from these systems, we propose a novel Leap Motion-based building block training system that incorporates system software capable of generating highly realistic virtual scenes, with the added capability of user behavior detection using Kinect. We conducted user testing of this new system, comparing the performance of a representative cohort with mild cognitive impairment (MCI) (n = 9) to that of disease-free participants (n = 10). Additionally, we conducted ergonomic experiments to assess the system’s performance in elderly people. The experimental results revealed significant differences between the MCI cohort and the control cohort. Specifically, the MCI cohort exhibited a reduced range of motion and longer task completion times compared to the control cohort. These findings have the potential to contribute to the differentiation of cognitive levels. In conclusion, our analysis of existing cognitive rehabilitation training systems provides valuable insights for researchers working on the development of future innovative cognitive rehabilitation training systems and enriches the non-pharmacological treatment models for cognitive impairment. Furthermore, the observed relationship between behavioral data, task completion times, and cognitive levels in older adults offers useful insights for the design of HCI-based approaches for diagnosing and assessing the treatment of MCI.