Human trust in machines is critical for effective human-machine interaction in virtual reality (VR). Prior work defined a three-layered framework of such trust but also indicated two deficiencies. Firstly, there is an absence of a model with metrics spanning all layers to objectively capture fluctuations of the trust (trust dynamics) in real time. Secondly, there is an inadequate consideration of human and machine reliability for the trust. Herein, this study proposed a trust model by defining metrics for all the layers and evaluated this model by considering human and machine reliability. Using objective and subjective data, the evaluation was based on two VR use-cases. The outcomes of the evaluation confirmed the pertinence of the model to capture trust dynamics in the presence of human and machine reliability. The objective data was notably more sensitive to capturing trust dynamics than the subjective counterpart. The model could enable designing trustworthy and adaptive VR.

Measuring cognitive workload (CWL) is crucial for dynamic task reallocation (i.e., adaptation) between a human and a machine in a human-machine system (HMS). A conventional measurement of the CWL is based on subjectively reported scores about the six factors of the NASA Task Load Index (NASA-TLX) questionnaire. The questionnaire cannot however capture real-time fluctuations of the factors for an objective quantification. Additionally, each of the factors is associated with distinct activities and can be influenced by individual characteristics and/or task contexts. Such HMS adaptation should thus consider the objective quantification of each factor. So far, the quantification remains largely unexplored, while existing studies reveal a potential use of an electroencephalography (EEG) in measuring the CWL levels (e.g., high, medium, and low). Herein, we presented a pioneering study to propose EEG features for quantifying the factors. The pertinence of the features was demonstrated by their strong correlations with the scores of the factors across three distinct cases of visuomotor tasks. The pertinence is the stepping stone toward factor-based interventions in enabling HMS adaptation.

Haptic perception of object shape is crucial for humans to interact with machines in human-machine systems (HMS). This perception is prone to disruptions arising from the human and/or machine sides of HMS. An unexplored topic is cognitive processes of the perception. Herein, this study examined the feasibility of measuring the cognitive processes within a virtual environment (i.e., an HMS). Non-invasive electroencephalography was employed to record brain activity of human participants during a task, which was perturbed by disruptions from the human and machine sides. The cognitive processes were measured by using an engagement ratio (ER) and an attention ratio (AR) as physiological metrics, besides behavioral metrics. The results of the study confirmed the feasibility of ER and AR to measure the processes and, in turn, opens an avenue towards elucidating the processes for improving HMS.

The assessment of mental workload from electroencephalogram (EEG) data for brain-computer interfaces (BCI) poses some challenges due to the interaction of spatial and temporal features within the data. Similar challenges are well known in the analysis of turbulent flows, which exhibit complex space-time correlations resulting from large-scale structures. The similarities motivated us to conduct this feasibility work of applying analytic methods of fluid dynamics – i.e., a scheme combining spectral proper orthogonal decomposition (SPOD) and spherical harmonics as basis functions – to EEG data for identifying features representing mental states. Based on EEG data of an existing BCI Hackathon, the scheme yielded some relevant features across subjects and sessions by relying only on a Fourier transform in time and the basis functions in space. The features were then classified by employing a conventional support vector machine algorithm to produce an accuracy comparable to those reported in a previous study on the same EEG data. This performance comparability indicates the scheme's potential for analyzing EEG data in BCI applications. Nevertheless, future work is needed to select specific features as general indicators of mental workload.

Brain machine interfaces (BMI) are becoming increasingly prevalent in diverse applications including motor rehabilitation, virtual reality training, etc. Two critical aspects of an effective BMI are machine reliability and cognitive workload (CWL). Previous studies have reported a notable effect of machine reliability on the 6 factors of the CWL. However, it remains unclear whether this effect can be detected in cognitive processes. Electroencephalography (EEG) is a widely used technique to explore cognitive processes by recording brain activities as signals. Therefore, we utilized the event-related spectral power (ERSP) feature of EEG signals to determine the cognitive processes regarding the effect of machine reliability. The results revealed that machine reliability affected the CWL factor of performance which was reflected in the$y$band activities of the right prefrontal cortex. The findings indicate the potential of cognitive processes in detecting the effect of machine reliability. The detection could pave the way for designing adaptive BMI to balance the machine reliability and the CWL.

Reliability is an important concept contributing to building trust in human-machine systems (HMS). Existing studies have reported separate assessment of human and machine reliability. Thus, there is a gap on considering human and machine reliability concurrently in HMS. To fill the gap, this study investigated the feasibility of such concurrent consideration by using a virtual environment (VE) approach to simulate an HMS. In a developed VE, each human participant performed a task of exploring an invisible surface to perceive its shape, followed by his/her response to a recommendation about the shape made by the VE setting (the machine). Related to human reliability, the perception might be disrupted through a mismatch between the actual shape and force feedback delivered to the participant’s hand. Associated with machine reliability, the recommendation could be incorrect to induce a fault in the setting. Thus, the shape of the invisible surface became an instrument to combine human and machine reliability. The outcomes of the study confirmed the feasibility of combining human and machine reliability in the HMS. Moreover, human reliability might be dominant in the HMS to accomplish the task.

Human-machine systems (HMS) need trustful cooperation between humans and machines for achieving a goal. Establishing such trust demands the machines’ adaptivity to the cognitive workload (CWL) of the humans. The CWL is conventionally measured as self-reported scores from a NASA-TLX questionnaire, susceptible to individual subjectivity. In contrast, logged brainwaves are useful for measuring the CWL objectively. However, there is a literature gap of mapping the brainwaves to a CWL factor - i.e., physical demand. As a feasibility, we thus proposed an activity ratio (AR) to measure the physical demand from the brainwaves. Statistical analyses indicated significant correlations between the AR and self-reported scores of the physical demand, compared to a well-known engagement ratio. This finding implied the feasibility of the AR to measure the physical demand.

Interactive human-machine systems (HMS), such as compute-based virtual environments (VEs), have been increasingly relied upon for decision-making. Building trust between human users and machines is crucial to enable a cooperative relationship. One aspect of building trust requires modeling sensory feedback from virtual objects in VEs to the users for appropriate understanding and utilization. In current VEs, of interest is modelling the integration of vibrotactile and force cues for providing sensory feedback to stimulate the haptic modality of the users. Behavioral models, such as maximum likelihood estimation, have failed to interpret the integration. Underlying this failure might be subtle internal processes of the human brain. Hence, we conducted an experiment to investigate the feasibility of modeling the integration using a drift-diffusion model (DDM), which is known to bridge observed behavioral outcomes and internal processes. In the experiment, human participants undertook a navigation and detection task within a 3D VE. Their task execution was aided by vibrotactile or/and force cues. Analyses on task accuracy and response time to the cues confirmed that DDM was feasible to interpret behavioral outcomes of the participants. The interpretation implies a link between the outcomes and the internal processes, paving a potential way to use DDM for elucidating the integration of vibrotactile and force cues.

A model of integration for vibrotactile and force cues is important for facilitating human users' task performance in human-machine systems. One of such human-machine systems is an interactive three-dimensional (3D) virtual environment (VE). In this paper, we proposed proportional likelihood estimation (PLE) as a model of integration for vibrotactile and force cues. Assuming human responses to cues as Gaussian distributions, PLE integrates these cues proportionally according to certain weighted contributions. We conducted an experiment to verify the suitability of PLE. For the experiment, we created a VE in which a human user executed interactively an identification task. The task required the user to identify visually indiscernible defects on a transmission line with a flying drone. The defects were indicated to the user through vibrotactile and/or force cues. These cues were in a co-located or dis-located setting, respectively, on the user's right hand and/or forearm. The PLE predictions of integrating the vibrotactile and force cues were able to match the empirical observation of these combined cues. PLE also elucidated this cue integration successfully when applying to an existing dataset acquired under a different experimental condition. Further analyses revealed that the cue integration may not be entirely additive. Hence, PLE could shed a light on the cue integration for facilitating user interaction in human-machine systems, like VEs.

It is recognized that system trustworthiness is a hyperstructure embodied by the structural, behavioral, and system dimensions with a set of coherent attributes. We explore a theoretical framework of tripartite trustworthiness that can be applied to real-world autonomous systems. We present a formal study of the essences and mathematical models of system trustworthiness and their quantitative measurements in the contexts of autonomous and mission-critical intelligent systems where humans and machines interact in a hybrid environment.

Efforts exist to combine a brain-machine interface (BMI) into a 3D virtual environment (VE) for visual tasks. User interaction via haptic stimuli within the VE is still unexplored for developing the BMI however, due to little understanding of cognitive processes related to such haptic interaction. Hence, we investigated a feasibility of estimating cognitive processes related to haptic interaction. Involved in the investigation, human participants undertook a task via different haptic stimuli (e.g., force and vibration) within a 3D VE . Their brain activities evoked by the stimuli were acquired as electroencephalography signals. Patterns of event-related potential and power spectral density were extracted from the signals, indicating activation in certain brain areas. The estimation of connectivity among these areas used directed transfer function, emphasizing on the middle of the β band (1/030 Hz) in the signals. The emphasis was due to the band's association with active focus and thinking. We found that, while behavioral differences were unapparent, all vibration-related stimuli yielded distinct active brain areas and connectivity to form certain cognitive processes. The finding implied a potential of localizing the processes for BMI-based haptic interaction.

Current virtual environments (VE) enable perceiving haptic stimuli to facilitate 3D user interaction, but lack brain-interfacial contents. Using electroencephalography (EEG), we undertook a feasibility study on exploring event-related potential (ERP) patterns of the user's brain responses during haptic interaction within a VE. The interaction was flying a virtual drone along a curved transmission line to detect defects under the stimuli (e.g., force increase and/or vibrotactile cues). We found that there were variations in the peak amplitudes and latencies (as ERP patterns) of the responses at about 200 ms post the onset of the stimuli. The largest negative peak occurred during 200~400 ms after the onset in all vibration-related blocks. Moreover, the amplitudes and latencies of the peak were differentiable among the vibration-related blocks. These findings imply feasible decoding of the brain responses during haptic interaction within VEs.

A well-designed collaboration between visual and automated analyses can facilitate complex tasks performed by an analyst (l.e., user). One such task is the study of spatiotemporal (unsteady) flow fields represented by velocity vectors. Our earlier work introduced a cluster-based technique of abstracting velocity patterns for visual analysis of flows. Though these patterns convey important information, the visual analysis overloads the human cognitive abilities of identifying and tracking patterns in space and time. To address this overloading, we have injected new spatial patterns into the visual analysis and developed an automated analysis to detect the temporal changes in the velocity and spatial patterns. For aiding the user's understanding of the complex relations between the spatial and temporal characteristics of the patterns, this paper presents a collaboration between the visual and automated analyses. To assess this collaboration, we have proposed a usefulness metrics to encompass usability and utility of an analysis process. Using two complex flow datasets with multiple actuations and thousands of time instants, we have conducted a preliminary assessment of the collaboration based on this metrics. The outcomes of the assessment indicate that the collaboration aids an analyst in identifying and tracking pattern changes. Therefore, the collaboration shows potential in facilitating the study of complex flow patterns.

In a three-dimensional (3D) virtual environment (VE), proper collaboration between vibrotactile and force cues - two cues of the haptic modality - is important to facilitate task performance of human users. Many studies report that collaborations between multi-sensory cues follow maximum likelihood estimation (MLE). However, an existing work finds that MLE yields a mean and an amplitude mismatches when interpreting the collaboration between the vibrotactile and force cues. We thus proposed mean-shifted MLE and conducted a human study to investigate the mismatches. For the study, we created a VE to replicate the visual scene, the 3D interactive task, and the cues from the existing work. Our participants were biased to rely on the vibrotactile cue for their tasks, departing from unbiased reliance on both cues in the existing work. Assessments of task completion time and task accuracy validated the replication. We found that based on task accuracy MLE explained the cue collaboration to certain degrees, agreed with the existing work. Mean-shifted MLE remedied the mean mismatch, but maintained the amplitude mismatch. Further examinations revealed that the collaboration between both cues may not be entirely additive. This sheds an insight for proper modeling of the collaboration between the vibrotactile and force cues to aid interactive tasks in VEs.

Robot-assisted surgical systems aim at enhancing surgeon's skills. Nonetheless, the learning curve for mastering such systems is very slow due to the motion-coupling mode that is usually presented in these systems for manipulating a surgical tool. This mode has limitations compared to the direct mapping mode used in open surgery for manipulating a tool. Virtual reality (VR) surgical simulators may reduce the learning time for transferring the surgeon's skills from direct mapping to motion coupling of tool manipulation. This may be accomplished by adding two features to the simulator. First, force models of tool-tissue interaction can be implemented in the haptic interface of the simulator. Second, VR-based surgical tasks can be designed to recreate directmapping mode and motion coupling mode of tool manipulation, as in open and robot-assisted surgeries, respectively. This may permit to transfer the surgeon's skills from open surgery to robotassisted surgery in a timely manner. This work presents a preliminary study on the effect of direct mapping mode and motion coupling mode of tool manipulation on the performance of naïve participants for VR-based brain tissue dissection. An Analytic force model of soft-tissue dissection was implemented in the simulator along with visual feedback of a predefined tool speed of 1 mm/s, which is observed in neurosurgery. The outcomes indicated that the motion quality of the tool via direct mapping was significantly better than with motion coupling. Thus, the study might serve as a first step toward the assessment of user's skills for VR-based robot-assisted dissection.

Vibrotactile and force cues of the haptic modality is increasing used to facilitate interactive tasks in three-dimensional (3D) virtual environments (VE). While maximum likelihood estimation (MLE) explains the integration of multi-sensory cues in many studies, an existing work yielded mean and amplitude mismatches when using MLE to interpret the integration of vibrotactile and force cues. To investigate these mismatches, we proposed mean-shifted MLE and conducted a study of comparing MLE and mean-shift MLE. Mean-shifted MLE shared the same additive assumption of the cues as MLE, but took account mean differences of both cues. In a VE, the study replicated the visual scene, the 3D interactive task, and the cues from the existing work. All human participants in the study were biased to rely on the vibrotactile cue for their task, departing from unbiased reliance towards both cues in the existing work. After validating the replications, we applied MLE and mean-shifted MLE to interpret the integration of the vibrotactile and force cues. Similar to the existing work, MLE failed to explain the mean mismatch. Mean-shifted MLE remedied this mismatch, but maintained the amplitude mismatch. Further examinations revealed that the integration of the vibrotactile and force cues might violate the additive assumption of MLE and mean-shifted MLE. This sheds a light for modeling the integration of vibrotactile and force cues to aid 3D interactive tasks within VEs.

Significant research efforts have been devoted to the development of force models that estimate soft-tissue biomechanical responses, finding an application on virtual reality (VR) based surgery training simulation. Nonetheless, the effects of force models on user performance of surgical tasks at different translation speeds are yet unclear. Thus, this work evaluated the effects of simple Weibull and realistic Analytic force models on 10 naïve human subjects for performing 1 degree-of-freedom (DOF) brain-tissue dissection tasks on a VR simulator at speeds of 0.10, 1.27, and 2.54 cm/s. Relying on 4 objective and 5 subjective performance metrics, two-way and one-way ANOVA analyses showed that a realistic force model such as the Analytic model is required to lessen the workload perceived by users only at a low dissection speed of 0.10 cm/s like that observed in neurosurgery. It was also found that dissections performed at the speed of 0.10 cm/s demand more refined manual skills than those at higher speeds. This finding complies with the lengthy surgery training curricula required to master surgical skills.

Multiuser collaboration within virtual environments (VEs) need effective means of communication. A real-world collaboration benefits from verbal and nonverbal communication channels. To promote the nonverbal communication within VEs, some research studies have explored various forms of vibrotactile cues that are either spatially co-located with an interaction device held by a user's hand, or dislocated from the device. These studies are focused on VEs that are established on a leader and a follower paradigm. A multiuser collaborative VE is however governed by an interaction model to handle the simultaneous interactive commands issued by multiple peer users. Proposed in our earlier work, the dynamic priority model (DP) is an interaction model that yields perceived equality in interaction and promotes the multiuser collaboration when peer users communicate through verbal dialogue. As a key means of nonverbal communication in VEs, co-located and dislocated vibrotactile cues might affect the perceived equality in interaction, and how users collaborate under the DP model. In this study, we have thus investigated the role of vibrotactile cues on multiuser collaboration under the DP model. We undertook this investigation in two cue settings: co-located and dislocated. We observed that the DP model yields perceived equality in interaction both in the absence and presence of vibrotactile cues. Also, the co-located cues significantly enhanced the multiuser collaboration compared to the dislocated cues. These observations imply a potential application of co-located vibrotactile cues to enhance multiuser collaboration within VEs.

Visualization of fluid flows at a high-Reynolds number (Re ~ 105) presents difficulties for user comprehension due to density and ambiguous interactions between vortices. Prior work has used cluster-based reduced-order modelling (CROM) to analyze the wake of a High-Speed Train (HST) with Re = 86,000. In this paper, we present a novel surface visualization to convey the spatiotemporal changes undergone by clustered vortices in the HST wake. This visualization is accomplished through dimensional reduction of 3D volumetric vortices into 1D ridges, and physics-based feature tracking. The result is 3D surfaces visualizing the behavior of the vortices in the HST wake. Compared to conventional still-image representations, these surfaces allow the user to quickly compare and analyze the two shedding cycles identified via CROM. The spatiotemporal differences of the primary vortices in these shedding cycles provide analytic insight to influence the aerodynamics of the HST.

Research efforts have been devoted to extraction and visualization of vortices in an unsteady (turbulent) flow. Characterizing the behaviors of the flow, vortices are identifiable as regions using a vortex detector known as the lambda2-criterion. Isosurface visualization renders vortex regions based on a chosen isovalue. However, it is highly challenging to choose one isovalue suitable for visualizing vortex regions of the entire flow field. A solution is the approach of maxima score that localizes vortex regions identified by the lambda2-criterion based on similarity scores relative to local extrema. The approach is however sensitive to noise or floating-point errors in the flow, leading to clutter in vortex visualization. As a feasibility study, this paper presents a threshold estimation to overcome this sensitivity. The estimation involves clustering on local minimum differences in lambda2 scalar values derived from the gradient tensor of the velocity field, and yields multiple values of the threshold without user intervention. Tested on several flows in various size and Reynolds number, the results of the threshold estimation confirmed overcoming the sensitivity of the maxima score approach. This indicates a potential of the threshold estimation to improve the robustness of the approach for vortex extraction and visualization.

Recent research efforts show the benefits of using machine learning and interactive visualizations in data analytics. However, there is a void in the implementation of these techniques for the analysis of large and complex 4-dimentional (4D) unsteady flows. Hence, this paper presents an initial development of a virtual environment (VE) to fill this void. The VE has a two-layer architecture with different technologies running in the back and fore grounds. Using machine learning, the background layer implements clustering algorithms for abstracting spatiotemporal patterns of the flows like flow features, spatial regions and temporal phases. The outputs of the clustering algorithms are fed to the foreground layer for interactive selection, sorting and filtering of these patterns. The operations in the foreground make up our designed techniques of flow pattern analysis that aims to provide greater comprehension of 4D flow data. Running in the foreground, the virtual reality (VR) technologies of stereoscopic rendering and haptic feedback further enhance these analysis techniques. Thus, our work introduces a novel environment for the interactive analysis of cluster-based flow pattern abstractions.

Proper integration of sensory cues facilitates 3D user interaction within virtual environments (VEs). Studies showed that the integration of visual and haptic cues follows maximum likelihood estimation (MLE). Little effort focuses however on the mechanism of integrating force and vibrotactile cues. We thus investigated MLE's suitability for integrating these cues. Within a VE, human users undertook 3D interaction of navigating a flying drone along a high-voltage transmission line for inspection. The users received individual force or vibrotactile cues, and their combinations in collocated and dislocated settings. The users' task performance including completion time and accuracy was assessed under each individual cue and setting. The presence of the vibrotactile cue promoted a better performance than the force cue alone. This agreed with the applicability of tactile cues for sensing 3D surfaces, herein setting a baseline for using MLE. The task performance under the collocated setting indicated a degree of combining the individual cues. In contrast, the performance under the dislocated setting was alike under the individual vibrotactile cue. These observations imply a possible role of MLE in integrating force and vibrotactile cues for 3D user interaction within VEs.

Collaborative virtual environments (VEs) require interaction models for resolving conflicts and promoting multiuser collaboration. Common models, such as the first-come-first-serve (FCFS) model, which grants interaction opportunities to the most agile user, and the static priority model, which gives interaction opportunities to the user with the highest predefined priority, disregard the importance of perceiving equality in interaction (EII) among all users. One exception is the dynamic priority (DP) model, as proposed in our earlier work, which grants interaction opportunities to a user based on the recency of his/her gained opportunities. To date, few research efforts have investigated the effect of interaction models on multiuser satisfaction. This paper hence presents an assessment of the DP model's effect on multiuser satisfaction within a collaborative VE. We first verified that the DP model allowed multiple users to perceive EII. We then conducted an experiment to examine the effect of the DP and FCFS models on multiuser satisfaction under a quasi-practical scenario that mimicked a decision-making meeting of experts. The framework of the examination was based on several metrics, which we proposed for the components of the ISO/IEC 25010:2011 standard. This framework resolved issues with existing metrics that measure user satisfaction by analyzing individual experience, thus omitting EII desired by multiple users. The results of the experiment indicated that the DP model fulfilled the metrics of the framework significantly better than the FCFS model. This observation implies a potential application of the DP model in collaborative VEs where multiuser satisfaction is the key to productive collaboration.

Multi-user efficacy is a key factor of genuine collaboration among multiple users towards a common goal. To assess multi-user efficacy, social scientists have traditionally applied subjective measurements from a theoretical perspective. Researchers in human-computer interaction have developed combined metrics of objective and subjective measurements. Nevertheless, the combined metrics fall short to fully cover the theoretical perspective of social scientists. To remedy this shortfall, we have developed a set of objective and subjective metrics to complete the theoretical perspective. Utilizing the metrics, we present in this paper a study to verify the robustness of our dynamic priority (DP) model, which under a quasi-practical scenario resolves command conflicts and promotes perceived equality in interaction among multiple users. In the study, we utilized a realistic scenario which differs from the quasi-practical scenario in the allowance of verbal communication among users. The results of the study revealed that the DP model yielded a significantly higher degree of multi-user efficacy under the realistic scenario than the quasi-practical scenario. Moreover, there was no significant difference of the perceived equality in interaction between both scenarios. These observations confirm the robustness of the DP model, and imply the potential application of the model for genuine collaboration within multi-user VEs.

Visualization of flow features, such as vortices, aids in analyzing complex unsteady (turbulent) flows and thus facilitate human cognition of flow phenomena. Typical data of unsteady flows are vector fields of velocity in four spatiotemporal dimensions (4D). Often empirical data of unsteady flows suffer from unknown measurement errors (i.e., undesired vectors). The undesired vectors lead to clutter in visualization. The clutter degrades information content and obstructs user interaction for analyses. As a feasibility study, this paper presents adaptive nonparametric approach (ANPA) to pre-process three-dimensional (3D) flow data at individual time frames. The pre-processing aimed to eliminate undesired vectors in the data. Tested on 4 empirical datasets with different intensities of vortex shedding, ANPA reduced undesired vectors, and thereby visual clutter, from the flow data without distorting vortices. The reduction was quantifiable by the decreased number of false positives in vortex identification. These observations indicate a potential to extend ANPA to pre-process 4D flow data for vortex visualization.

Both surgical simulation and robot-assisted surgery require haptic models of tool-tissue interaction for force rendering. Most efforts of haptic modeling have focused on characterizing tool-tissue interaction of soft-tissue indentation, insertion and cutting. Less attention has been devoted to soft-tissue dissection however. For the dissection, haptic models remain elusive to meet two requirements as: to represent nonlinearity of soft-tissue responses and to comply with the time constraint of 1 ms for force rendering. Hence, this paper presents a modeling framework towards developing an analytic haptic model for force rendering of the dissection. Based on estimation theories, the framework devises an analytic model to approximate an empirical force-distance profile of the dissection. Applying the framework to 2 different empirical profiles as use cases, the derived models estimated about 72% and 91% of the empirical data, respectively. Algorithm implementation of these models in Matlab yielded a computational time of about 24 μs, much less than 1 ms. The outcomes indicate a potential of using the framework to develop an analytic haptic model for force rendering of soft-tissue dissection.

Effective extraction of characteristic structures of flow features, such as vortices, can simplify visualization and thus potentially enable users easier cognition of flow field data. In this paper we present a novel method to extract vortex structures through correlation of flow domains transformed from the velocity field. This correlation is based upon our earlier "maxima score" approach, which is domain-independent and normalized. By correlating the maxima score of flow domains, we have improved its capabilities for extracting characteristic structures of vortices. We have compared our method with the feedback of flow analysis experts, and verified the capabilities of our method using flow data derived from computational fluid dynamics and experimental measurements. These results show that this new method reduces noise, and operates at a higher sensitivity for vortex extraction than the maxima score is capable of alone.

Multi-user usability of collaborative virtual environments (VEs) require the consideration of users’ socio-human needs. However, most investigations of the usability have focused on either the technologies involved or individual user experience. Few have examined the effect of interaction models o

In robot-assisted surgery, a critical factor is to handle pose (position and orientation) mismatch between a stylus-style hand controller and a surgical tool (robot-assisted tool) attached to the end-effector of a robot. The mismatch has similar characteristics as robot calibration in industrial settings. Nevertheless, any methods for correcting the mismatch need to meet certain computation and accuracy requirements, which are derived from the constraints of a robot-assisted surgical system. On a virtual reality simulator, we use a haptic device (PHANToM Premium 1.5/6DOF) as the hand controller to actuate a robot-assisted surgical tool via neuroArm - a robotic system in microscopic neurosurgery. Within the workspace of the tool, we have defined the computation and accuracy requirements of correction as 1.0 ms and 30.0 μm, respectively. Towards the correction of the pose mismatch, this current work first assesses the suitability of the Newton-Raphson (NR) method for addressing the position mismatch between the haptic interface point (HIP) of the haptic device and the tooltip of the surgical tool. For fast computation, we have modified the NR method to take advantage of its quadratic rate of convergence. This modification adds a feedback loop for selecting appropriate initial values. As well, we have verified the non-singularities of the workspace where the position mismatch needs to be corrected. Assessed in the workspace of 90 targets, the modified NR method achieves an accuracy between the HIP and the tooltip at about 1.0 μm in less than 64 μs - meeting both requirements of correction. Thus, this work confirms the suitability of the modified NR method to efficiently correct the position mismatch between the HIP and the tooltip on neuroArm.

Multi-user collaborative virtual environments (VEs) need strategies of conflict resolution to handle simultaneous interaction with shared objects. Current strategies are first-come-first-serve (FCFS) and predefined static priority of each user. These strategies cannot provide each user with a perc

In robot-assisted surgery, haptic devices as hand controllers play an important role in the surgeon's ability of manipulating tissues. Most research activities have focused on providing force feedback as haptic information. Considering a circular tracing task, we presented two models of torque feedback. On a Virtual Reality simulator of neuroArm, a robotic system for microscopic neurosurgery, we investigated the suitability of these models for rendering torque feedback of each rotational degree-of-freedom. Without sacrificing translational and rotational motion ranges, we verified that the device PHANToM Premium 1.5/6DOF is adequate as a hand controller of neuroArm to replace the modified version of PHANToM Premium 1.5 interfacing originally with neuroArm. Using the device PHANToM Premium 1.5/6DOF, our preliminary results yielded a suitability of both models of torque feedback for performing a circular tracing task assisted by neuroArm.

Virtual Reality (VR) technology has great potential as a tool for training. Remote haptic virtual environments (VE) allow multiple users to interact in the same virtual space to accomplish tasks. Combining these ideas, it is possible to apply a remote VE to connect skilled trainers with remote trainees for training. In this paper we examined guided haptic training as an interaction method where one user guides another user through a VR simulation. We considered interaction with virtual soft objects, which were pressed and deformed. Within a remote VE for guided haptic training, network factors such as delay could influence how the guided user perceives object softness. In this paper, we examined how incongruent delays between the streams of visual and haptic information affect this perception. We found that when haptic information was delayed 133.33 ms beyond visual information there was a statistically significant difference in the perception of object softness. These preliminary observations can be used to locate a threshold for applying to future remote guided VE training tools.

Being able to work effectively in any collaborative venture is beneficial for everyone who is involved in the undertaking. In order for a collaborative venture to be effective, it must allow a wide range of people possessing different expertise to interact in a seamless manner. Thus, it is of utmost importance to investigate the underlying strategy that permits people with different expertise to work together and accomplish the task at hand. We began our investigation by observing, from previous research, that the human visual system comprises of two streams; each stream possessing complementary function. These streams are able to work flawlessly allowing people to perform visually guided actions with the slightest effort. Thus, in this pilot study we aim to investigate how the neural mechanism of the human visual system can be applied to help two people to collaborate in a seamless manner. In particular, we explore how the neural mechanisms of the human visual system could be used to allow two people to complete a collaborative task without the use of any verbal communication. As a result, this preliminary study revealed some intrinsic characteristics of the underlying strategy that makes any collaborative venture effective.

In this paper, we report the first step of our ongoing research toward the creation of an intuitive and interactive environment for manipulating and analyzing geological datasets. This first step of our project aimed at the development of a manipulation system through the employment of haptic sense and gesture detection, into a virtual environment. The developed prototype integrates stereoscopic interactive visual rendering, haptic feedback and real-time hand tracking via a range camera, in a multithreaded architecture. Our prototype has been tested with both a synthetic 3D terrain and geological datasets. Our first results confirm the effectiveness of the selected architecture.

A current issue in human-computer interaction is the design of haptic stimuli. Recent studies reported that humans can successfully recognize various haptic stimuli, and thus suggest methods of designing distinguishable haptic stimuli. However, such methods have been complicated and inconsistent, indicating a need for exploring different parameters associated with human perception. Therefore we conducted this pilot study on the distinguishability of haptic stimuli. We took advantage of frequency domain analysis to explore the potential of a parameter of relative percent power difference (%PD). The Fourier series was used to design a range of synthetic haptic stimuli using approximations of square and saw-tooth signals of the same fundamental frequency and amplitude. The stimuli differed by the range of harmonic components in each series. Preliminary results revealed that stimuli based on the saw-tooth signal were more distinguishable than their square based counterparts. While investigating the parameter of %PD to measure differentiability, we observed that participants had more difficulty in distinguishing stimuli with smaller relative %PD than stimuli with greater relative %PD. A considerable change of differentiability between 10 and 35 %PD pointed towards potential just-noticeable-difference for distinguishability. Further investigation is needed to support these findings.

Soft tissue palpation plays an important role in diagnosing various diseases. Palpating skills are tedious to learn due to the difficulty of describing the sense of touch. Because of its interactive nature, a virtual reality (VR) training system embedding with real-time soft-tissue models may be helpful to teach such skills to medical residents. Studies show that such a VR system impacts human perception during palpating at various levels, largely due to the real-time models. Therefore, we propose a formal method for evaluating real-time models considering the human perception. Based upon surface (multi-vertex) contact with 4 force distributions, the evaluation compared a real-time model with a Finite Element Method (FEM) model featuring physical parameters. The comparison consisted of two statistical approaches - ANOVA and Bland and Altman agreement- to assess both visual displacement and force feedback. A case study demonstrated the advantages provided by this evaluation method.

Virtual reality (VR) has been gaining popularity in surgical planning and simulation. Most VR surgical simulation systems provide haptic (pertinent to the sense of touch) and visual information simultaneously using certain alignments between a haptic device and visual display. A critical aspect of such VR surgical systems is to represent both haptic and visual information accurately to avoid perceptual illusions (e.g., to distinguish the softness of organs/tissues). This study compared three different alignments (same-location alignment, vertical alignment, and horizontal alignment) between a haptic device and visual display that are widely used in VR systems. We conducted three experiments to study the influence of each alignment on the perception of object softness. In each experiment, we tested 15 different human subjects with varying availability of haptic and visual information. During each trial, the task of the subject was to discriminate object softness between two deformable balls in different viewing angles. We analyzed the following dependent measurements: subject perception of object softness and objective measurements of maximum force and maximum pressing depth. The analysis results reveal that all three alignments (independent variables) have similar effect on subjective perception of object softness within the interval of viewing angles from -7.5°to +7.5°. The viewing angle does not affect objective measurements. The same-location alignment requires less physical effort compared with the other two alignments. These observations have implications in creating accurate simulation and interaction for VR surgical systems.

A critical issue of virtual reality (VR) surgical systems is to correctly represent both haptic and visual information for distinguishing the softness of organs/tissues. We investigated the relationship between subjective perception of object softness and objective measurements of haptic and visual information. On a co-location VR setup, human subjects pressed deformable balls (simulating organs/tissues) under the conditions of both haptic and visual information available and only haptic (or visual) information available. We recorded and analyzed the subject's selection (subjective perception) of the harder object between two balls and objective measurements of maximum force (haptic) and pressing depth (visual). The results preliminarily indicated that subjective perception behaves differently from objective measurements in perceiving object softness. This has implications for creating accurate simulation in VR surgical systems.

This paper reviews empirical work on human visuomotor systems and describes the constraints that limit the transformations these systems can perform. A tentative model is put forward, which describes how information might flow from visual input to motor output. The purpose of this paper is to show how visual neuroscience and robotics can inform each other.

The authors investigated the kinematics of human visually guided prehension in a situation similar to that typically used in visual servo control of autonomous robots. They found that kinematic parameters of the human grasp, such as transport velocity, path, and grip aperture, were determined by the 3-dimensional geometric structure of the target object, not the 2-dimensional projected image of the object. The results of this study have important implications for the design of reliable and efficient control systems for robots with human-like capabilities.