Measuring cognitive workload (CWL) is crucial for dynamic task reallocation (i.e., adaptation) between a human and a machine in a human-machine system (HMS). A conventional measurement of the CWL is based on subjectively reported scores about the six factors of the NASA Task Load Index (NASA-TLX) questionnaire. The questionnaire cannot however capture real-time fluctuations of the factors for an objective quantification. Additionally, each of the factors is associated with distinct activities and can be influenced by individual characteristics and/or task contexts. Such HMS adaptation should thus consider the objective quantification of each factor. So far, the quantification remains largely unexplored, while existing studies reveal a potential use of an electroencephalography (EEG) in measuring the CWL levels (e.g., high, medium, and low). Herein, we presented a pioneering study to propose EEG features for quantifying the factors. The pertinence of the features was demonstrated by their strong correlations with the scores of the factors across three distinct cases of visuomotor tasks. The pertinence is the stepping stone toward factor-based interventions in enabling HMS adaptation.

Human-machine systems (HMS) need trustful cooperation between humans and machines for achieving a goal. Establishing such trust demands the machines’ adaptivity to the cognitive workload (CWL) of the humans. The CWL is conventionally measured as self-reported scores from a NASA-TLX questionnaire, susceptible to individual subjectivity. In contrast, logged brainwaves are useful for measuring the CWL objectively. However, there is a literature gap of mapping the brainwaves to a CWL factor - i.e., physical demand. As a feasibility, we thus proposed an activity ratio (AR) to measure the physical demand from the brainwaves. Statistical analyses indicated significant correlations between the AR and self-reported scores of the physical demand, compared to a well-known engagement ratio. This finding implied the feasibility of the AR to measure the physical demand.

Interactive human-machine systems (HMS), such as compute-based virtual environments (VEs), have been increasingly relied upon for decision-making. Building trust between human users and machines is crucial to enable a cooperative relationship. One aspect of building trust requires modeling sensory feedback from virtual objects in VEs to the users for appropriate understanding and utilization. In current VEs, of interest is modelling the integration of vibrotactile and force cues for providing sensory feedback to stimulate the haptic modality of the users. Behavioral models, such as maximum likelihood estimation, have failed to interpret the integration. Underlying this failure might be subtle internal processes of the human brain. Hence, we conducted an experiment to investigate the feasibility of modeling the integration using a drift-diffusion model (DDM), which is known to bridge observed behavioral outcomes and internal processes. In the experiment, human participants undertook a navigation and detection task within a 3D VE. Their task execution was aided by vibrotactile or/and force cues. Analyses on task accuracy and response time to the cues confirmed that DDM was feasible to interpret behavioral outcomes of the participants. The interpretation implies a link between the outcomes and the internal processes, paving a potential way to use DDM for elucidating the integration of vibrotactile and force cues.

A model of integration for vibrotactile and force cues is important for facilitating human users' task performance in human-machine systems. One of such human-machine systems is an interactive three-dimensional (3D) virtual environment (VE). In this paper, we proposed proportional likelihood estimation (PLE) as a model of integration for vibrotactile and force cues. Assuming human responses to cues as Gaussian distributions, PLE integrates these cues proportionally according to certain weighted contributions. We conducted an experiment to verify the suitability of PLE. For the experiment, we created a VE in which a human user executed interactively an identification task. The task required the user to identify visually indiscernible defects on a transmission line with a flying drone. The defects were indicated to the user through vibrotactile and/or force cues. These cues were in a co-located or dis-located setting, respectively, on the user's right hand and/or forearm. The PLE predictions of integrating the vibrotactile and force cues were able to match the empirical observation of these combined cues. PLE also elucidated this cue integration successfully when applying to an existing dataset acquired under a different experimental condition. Further analyses revealed that the cue integration may not be entirely additive. Hence, PLE could shed a light on the cue integration for facilitating user interaction in human-machine systems, like VEs.

Efforts exist to combine a brain-machine interface (BMI) into a 3D virtual environment (VE) for visual tasks. User interaction via haptic stimuli within the VE is still unexplored for developing the BMI however, due to little understanding of cognitive processes related to such haptic interaction. Hence, we investigated a feasibility of estimating cognitive processes related to haptic interaction. Involved in the investigation, human participants undertook a task via different haptic stimuli (e.g., force and vibration) within a 3D VE . Their brain activities evoked by the stimuli were acquired as electroencephalography signals. Patterns of event-related potential and power spectral density were extracted from the signals, indicating activation in certain brain areas. The estimation of connectivity among these areas used directed transfer function, emphasizing on the middle of the β band (1/030 Hz) in the signals. The emphasis was due to the band's association with active focus and thinking. We found that, while behavioral differences were unapparent, all vibration-related stimuli yielded distinct active brain areas and connectivity to form certain cognitive processes. The finding implied a potential of localizing the processes for BMI-based haptic interaction.

Current virtual environments (VE) enable perceiving haptic stimuli to facilitate 3D user interaction, but lack brain-interfacial contents. Using electroencephalography (EEG), we undertook a feasibility study on exploring event-related potential (ERP) patterns of the user's brain responses during haptic interaction within a VE. The interaction was flying a virtual drone along a curved transmission line to detect defects under the stimuli (e.g., force increase and/or vibrotactile cues). We found that there were variations in the peak amplitudes and latencies (as ERP patterns) of the responses at about 200 ms post the onset of the stimuli. The largest negative peak occurred during 200~400 ms after the onset in all vibration-related blocks. Moreover, the amplitudes and latencies of the peak were differentiable among the vibration-related blocks. These findings imply feasible decoding of the brain responses during haptic interaction within VEs.

In a three-dimensional (3D) virtual environment (VE), proper collaboration between vibrotactile and force cues - two cues of the haptic modality - is important to facilitate task performance of human users. Many studies report that collaborations between multi-sensory cues follow maximum likelihood estimation (MLE). However, an existing work finds that MLE yields a mean and an amplitude mismatches when interpreting the collaboration between the vibrotactile and force cues. We thus proposed mean-shifted MLE and conducted a human study to investigate the mismatches. For the study, we created a VE to replicate the visual scene, the 3D interactive task, and the cues from the existing work. Our participants were biased to rely on the vibrotactile cue for their tasks, departing from unbiased reliance on both cues in the existing work. Assessments of task completion time and task accuracy validated the replication. We found that based on task accuracy MLE explained the cue collaboration to certain degrees, agreed with the existing work. Mean-shifted MLE remedied the mean mismatch, but maintained the amplitude mismatch. Further examinations revealed that the collaboration between both cues may not be entirely additive. This sheds an insight for proper modeling of the collaboration between the vibrotactile and force cues to aid interactive tasks in VEs.

Vibrotactile and force cues of the haptic modality is increasing used to facilitate interactive tasks in three-dimensional (3D) virtual environments (VE). While maximum likelihood estimation (MLE) explains the integration of multi-sensory cues in many studies, an existing work yielded mean and amplitude mismatches when using MLE to interpret the integration of vibrotactile and force cues. To investigate these mismatches, we proposed mean-shifted MLE and conducted a study of comparing MLE and mean-shift MLE. Mean-shifted MLE shared the same additive assumption of the cues as MLE, but took account mean differences of both cues. In a VE, the study replicated the visual scene, the 3D interactive task, and the cues from the existing work. All human participants in the study were biased to rely on the vibrotactile cue for their task, departing from unbiased reliance towards both cues in the existing work. After validating the replications, we applied MLE and mean-shifted MLE to interpret the integration of the vibrotactile and force cues. Similar to the existing work, MLE failed to explain the mean mismatch. Mean-shifted MLE remedied this mismatch, but maintained the amplitude mismatch. Further examinations revealed that the integration of the vibrotactile and force cues might violate the additive assumption of MLE and mean-shifted MLE. This sheds a light for modeling the integration of vibrotactile and force cues to aid 3D interactive tasks within VEs.

Proper integration of sensory cues facilitates 3D user interaction within virtual environments (VEs). Studies showed that the integration of visual and haptic cues follows maximum likelihood estimation (MLE). Little effort focuses however on the mechanism of integrating force and vibrotactile cues. We thus investigated MLE's suitability for integrating these cues. Within a VE, human users undertook 3D interaction of navigating a flying drone along a high-voltage transmission line for inspection. The users received individual force or vibrotactile cues, and their combinations in collocated and dislocated settings. The users' task performance including completion time and accuracy was assessed under each individual cue and setting. The presence of the vibrotactile cue promoted a better performance than the force cue alone. This agreed with the applicability of tactile cues for sensing 3D surfaces, herein setting a baseline for using MLE. The task performance under the collocated setting indicated a degree of combining the individual cues. In contrast, the performance under the dislocated setting was alike under the individual vibrotactile cue. These observations imply a possible role of MLE in integrating force and vibrotactile cues for 3D user interaction within VEs.