We propose an algorithm that generates a vibration, an impact, or a vibration+impact haptic effect by processing a sound signal in real time. Our algorithm is selective in that it matches the most appropriate type of haptic effects to the sound using a machine-learning classifier (random forest) that is built on expert-labeled datasets. Our algorithm is tailored to enhance user experiences for video game play, and we present two examples for the RPG (role-playing game) and FPS (first-person shooter) genres. We demonstrate the effectiveness of our algorithm by a user study in comparison to other state-of-the-art (SOTA) methods for the same cross-modal conversion. Our system elicits better multisensory user experiences than the SOTA algorithms for both game genres.

Robots can be used to mitigate risks in unsafe and austere settings. In recent years, explosive ordnance disposal robots have reduced the technician's time-on-target, and thus, reduce the direct risk of exposure. This article focuses on the study and development of innovative techniques as the foundational work for a new robot platform. The proposed system includes an organic electrochemical transistor device to detect the existence of explosive residues, and lead to decisions for safe-removal progress. Taurus' surgical gripper facilitates object tactile exploration, and manipulation with control precision to the millimeter range. The highly sensitive triboelectric tactile sensor could reduce intrusiveness during contact, and mitigate the risk of detonation. Haptic devices and visual displays are used to convey important signals, in order to improve the situational awareness of the teleoperator. A machine learning classifier can be used to assist the user to identify objects from tactile sampling. The integration of these methodologies allows for a sensitive approach to concealed objects that are only accessible through tactile sensing.

Stand-alone devices for tactile speech reception serve a need as communication aids for persons with profound sensory impairments as well as in applications such as human-computer interfaces and remote communication when the normal auditory and visual channels are compromised or overloaded. The current research is concerned with perceptual evaluations of a phoneme-based tactile speech communication device in which a unique tactile code was assigned to each of the 24 consonants and 15 vowels of English. The tactile phonemic display was conveyed through an array of 24 tactors that stimulated the dorsal and ventral surfaces of the forearm. Experiments examined the recognition of individual words as a function of the inter-phoneme interval (Study 1) and two-word phrases as a function of the inter-word interval (Study 2). Following an average training period of 4.3 hrs on phoneme and word recognition tasks, mean scores for the recognition of individual words in Study 1 ranged from 87.7% correct to 74.3% correct as the inter-phoneme interval decreased from 300 to 0 ms. In Study 2, following an average of 2.5 hours of training on the two-word phrase task, both words in the phrase were identified with an accuracy of 75% correct using an inter-word interval of 1 sec and an inter-phoneme interval of 150 ms. Effective transmission rates achieved on this task were estimated to be on the order of 30 to 35 words/min.

This is a survey article on the information transmission capability of haptic devices and ways of maximizing it. It is intended for readers who are engineering professionals interested in developing novel haptic interfaces for a variety of applications but are not necessarily trained in haptic science or human user research. We posit that the ultimate goal of any interface is the exchange of information between a machine and a user, and as such, the evaluation should involve estimating the information-transmission capability with human users. We conducted a literature survey on studies of haptic devices evaluated with human users using an information theoretic framework. Our goal was to discover and summarize best practices that can lead to high information transmission. The results confirmed findings from our own previous studies, uncovered new ways to effectively increase information transmission, and pointed to the need for broader dissemination of proper experimental methodology. We, therefore, present a concise yet comprehensive tutorial on psychophysical methodology for estimating information transfer (IT) and IT rate with humans, survey results on the typical IT achievable with haptic devices, and guidelines for maximizing information transmission with any human-machine interfaces. Although we focus on haptic systems, the information-theoretic framework, the psychophysical methods, and the guidelines presented in this article are applicable to other sensory modalities and multimodal interface systems also.

We investigate the effect of information content in sensory feedback on typing performance using a flat keyboard. We build a flat keyboard apparatus with haptic and auditory keyclick feedback. We evaluate and compare typing performance with key-press confirmation and key-correctness information through sensory feedback. Twelve participants are asked to touch-type a number of randomly selected phrases under various combinations of visual, auditory and haptic sensory feedback conditions. The results show that typing speed is not significantly affected by the information content in sensory feedback, but the uncorrected error rate is significantly lower when key-correctness information is available. The results also show that key-correctness information leads to more corrected errors and lowers typing efficiency. Our findings are useful for developing flat keyboards with assistive information through sensory feedback. Our study is the first step towards improving typing performance on flat keyboards by delivering more advanced and comprehensive assistive information beyond the visual channel.

The present study used a flat keyboard without moving keys and enabled with haptic keyclick feedback to examine the effect of haptic keyclick feedback on touch typing performance. We investigated, with well-controlled stimuli and a within-participant design, how haptic keyclick feedback might improve typing performance in terms of typing speed, typing efficiency and typing errors. Of the three kinds of haptic feedback we tested, all increased typing speed and decreased typing errors compared to a condition without haptic feedback. We did not find significant differences among the types of haptic feedback. We also found that auditory keyclick feedback alone is not as effective as haptic keyclick feedback, and the addition of auditory feedback to haptic feedback does not lead to any significant improvement in typing performance. We also learned that global haptic keyclick feedback simulated through local keyclick feedback on each key (as opposed to haptic feedback all over the keyboard) might have the additional and unexpected benefit of helping a typist to locate keys on a keyboard. Furthermore, the participants preferred auditory or haptic keyclick feedback to no feedback, and haptic feedback restricted to the typing finger alone is preferred to that over a larger area of the keyboard.

Managing time while presenting is challenging, but mobile devices offer both convenience and flexibility in their ability to support the end-to-end process of setting, refining, and following presentation time targets. From an initial HCI-Q study of 20 presenters, we identified the need to set such targets per 'zone' of consecutive slides (rather than per slide or for the whole talk), as well as the need for feedback that accommodates two distinct attitudes to time management. These findings led to the design of TalkZones, a mobile application for timing support. When giving a 20-slide, 6m40s rehearsed but interrupted talk, 12 participants who used TalkZones registered a mean overrun of only 8s, compared with 1m49s for 12 participants who used a regular timer. We observed a similar 2% overrun in our final study of 8 speakers giving rehearsed 30-minute talks in 20 minutes. Overall, we show that TalkZones can encourage presenters to advance slides before it is too late to recover, even under the adverse timing conditions of short and shortened talks.

We investigate the role of contact location information on the perception of local features during contour following in a virtual environment. An absolute identification experiment is conducted under force-alone and force-plus-contact-location conditions to investigate the effect of the contact location information. The results show that the participants identify the local features significantly better in terms of higher information transfer for the force-plus-contact-location condition, while no significant difference was found for measures of the efficacy of contour following between the two conditions. Further data analyses indicate that the improved identification of local features with contact location information is due to the improved identification of small surface features.

A novel, low inertia, two degree-of-freedom (2-DOF) contact location display (CLD) device has been designed, prototyped, and tested. This device positions a small spherical contactor beneath the user's fingerpad at the point of contact between the finger and a virtual surface. Kinesthetic forces are provided by a custom haptic device attached to the CLD. The contactor is remotely driven by push-pull wires to reduce the effective inertia of the device at the user's fingertip; however, this design results in significant mechanical backlash. This backlash is characterized and partially compensated for in software. An experiment was used to evaluate several methods of rendering tactile feedback, each using a different method of prepositioning the contactor. The results show no statistical performance differences between rendering conditions. However, a post-experiment survey shows that participants perceived contact location + kinesthetic feedback as more realistic than pure kinesthetic feedback.

Behavioral studies of multisensory integration and cross-modal spatial attention have identified many potential benefits of using interfaces that engage more than just a single sense in complex operating environments. Particularly relevant in terms of application, the latest research highlights that: 1) multimodal signals can be used to reorient spatial attention effectively under conditions of high operator workload in which unimodal signals may be ineffective; 2) multimodal signals are less likely to be masked in noisy environments; and 3) there are natural links between specific signals and particular behavioral responses (e.g., head turning). However, taking advantage of these potential benefits requires that interface designers take into account the limitations of the human operator. In particular, multimodal interfaces should normally be designed so as to minimize any spatial incongruence between component warning signals presented in different sensory modalities that relate to the same event. Building on this rapidly growing cognitive neuroscience knowledge base, the last decade has witnessed the development of a number of highly effective multimodal interfaces for driving, aviation, the military, medicine, and sports.

Haptics research is by definition a very interdisciplinary field, with scientists and engineers building and evaluating tactile and haptic displays and exploring the domains in which the technology developed can be most optimally used. Central to this process are human user studies in which the displays are evaluated in terms of their capacity to present information to users and the sensitivity of the user to changes in display parameters. A variety of psychophysical procedures are employed for this purpose, most of which have been adapted from research on the visual and auditory systems. The sensory systems involved in processing haptic information present unique challenges due to the active nature of sensory exploration and these need to be taken into account in designing psychophysical experiments with tactile and haptic displays. In this half-day tutorial we will give an overview of psychophysical methods that can be used to analyze haptic perception, including those associated with measuring thresholds, information transfer and multi-dimensional scaling. We will also consider how these methods should be employed in the context of studying novel tactile and haptic illusions.

The effect of contact location information on the perception of virtual edges was investigated by comparing human edge sharpness discrimination under the force-alone and force-plus-contact-location conditions. The virtual object consisted of the 2D profile of an edge with two adjoining surfaces. Edge sharpness JNDs for both conditions increased from about 2 to 7 mm as the edge radii increased from 2.5 to 20.0 mm, and no significant difference was found between the two conditions. A follow-up experiment with the contact-location alone condition resulted in higher (worse) edge sharpness discrimination thresholds, especially at higher edge radius values. Our results suggest that contact location cues alone are capable of conveying edge sharpness information, but that force cues dominate edge sharpness perception when both types of cues are available.

The present study investigated the use of visual trajectory and haptic force information in learning concepts involving force. Specifically, learning modules for instructing buoyant forces were developed for use with a computer monitor and a force feedback device. Students from an elementary school at the fourth and sixth grades were recruited to participate in the study. The students were separated into visual and visuohaptic groups to measure the possible benefits haptic feedback might provide as compared to the vision-alone condition. A 10-question content test was developed and administered before and after the learning activities. The pretest and posttest scores showed that all students benefited from the computer simulations. The visuohaptic group did not perform significantly better than the visual group. An important finding was that the fourth graders learned as much as the sixth graders, despite their younger age and little prior exposure to concepts such as density and volume, which are important for understanding buoyancy. Future work will design instructional and assessment materials that focus more on the haptic modality.

Past studies have established a crossmodal spatial attentional link among vision, audition, and touch. The present study examined the dependence of visual attention on the distance between a distal visual target (a changing element among static distractors) and the quadrant of the visual display cued by a proximal tactile stimulus. The distance between the center of the cued visual quadrant and the visual target was one of six values: 0, 90, 180, 350, 450, and 550 pixels. The distances of 0, 90, and 180 corresponded to the valid tactile cueing condition, where the tactile cue and the visual target occurred in the same quadrant. The distances of 350, 450, and 550 corresponded to the invalid tactile cueing condition, where the tactilely-cued quadrant did not match that of the visual change. Results from 10 young adults showed that mean response time increased with respect to the cue-target distance, thereby confirming a gradient of visual attention for proximal-distal tactile cueing. In addition, the response times for valid tactile cues were shorter than those for invalid tactile cues, confirming earlier findings that valid tactile cues facilitate visual search and invalid tactile cues interfere with visual search. The findings of the present study have implications for the design of multimodal attention-cueing systems for practical applications such as collision warning systems in automobiles.

We present a method to improve watermark robustness by exploiting the masking effect of surface roughness on watermark visibility. Our idea is to adapt watermark strength to local surface roughness based on the knowledge that human eyes are less sensitive to changes on a rougher surface patch than those on a smoother surface. In order to quantify human sensitivity to surface roughness of polygonal meshes, we conducted a rigorous psychovisual experiment to obtain human watermark detection thresholds as a function of surface roughness. The results can be used to adaptively select watermark strength according to local surface roughness during the watermark embedding process. To test our idea, we applied it to the modified versions of two popular 3-D watermarking methods, one proposed by Benedens and one by Cayre and Macq. Experimental results showed that our approach improves watermark robustness as compared to the original algorithms. Further analyses indicated that the average watermark strength allowed by our roughness-adaptive method was larger than that by the original Benedens's and Cayre and Macq's methods while ensuring watermark imperceptibility. This was the main reason for the improved robustness observed in our experiments. We conclude that exploiting the masking property of human vision is a viable way to improve the robustness of 3-D watermarks, and can potentially be applied to other 3-D digital watermarking techniques.

Previous research on crossmodal attentional orienting has reported speeded reaction times (RT) when the stimuli from the different modalities are in the same spatial location and slowed RTs when the stimuli are presented in very different locations (e.g., opposite sides of the body). However, little is known about what occurs for spatial interactions between these two extremes. We systematically varied the separation between cues and targets to quantify the spatial distribution of crossmodal attention. The orthogonal cueing paradigm [Spence et al. 1998] was used. Visual targets presented above or below the forearm were preceded by either vibrotactile cues presented on the forearm, auditory cues presented below the forearm, or visual cues presented on the forearm. The presentation of both unimodal and crossmodal cues led to a roughly monotonic increase in RT as a function of the cue-target separation. Unimodal visual cueing resulted in an attentional focus that was significantly narrower than that produced by crossmodal cues: the distribution of visual attention for visual cues had roughly half of the lateral extent of that produced by tactile cueing and roughly one fourth of the lateral extent as that produced by auditory cueing. This occurred when both seven (Experiment 1) and three (Experiment 2) cue locations were used suggesting that the effects are not primarily due to differences in the ability to localize the cues. These findings suggest that the location of tactile and auditory warning signals does not have to be controlled as precisely as the location of visual warning signals to facilitate a response to the critical visual event.

The authors report six experiments on the human ability to discriminate and identify finger joint-angle positions using active motion. The PIP (proximal interphalangeal) joint of the index finger was examined in Exps. 1--3 and the MCP (metacarpophalangeal) joint in Exps. 4--6. In Exp. 1, the just noticeable difference (JND) of PIP joint-angle position was measured when the MCP joint was either fully extended or halfway bent. In Exp. 2, the JND of PIP joint-angle position as a function of PIP joint-angle reference position was measured when the PIP joint was almost fully extended, halfway bent, or almost fully flexed. In Exp. 3, the information transfer of PIP joint-angle position was estimated with the MCP joint in a fully extended position. In Exps. 4--6, the JND and the information transfer of MCP joint-angle position were studied with a similar experimental design. The results show that the JNDs of the PIP joint-angle position were roughly constant (2.5°−2.7°) independent of the PIP joint-angle reference position or the MCP joint-angle position used (Exps. 1 and 2). The JNDs of the MCP joint-angle position, however, increased with the flexion of both the PIP and MCP joints and ranged from 1.7° to 2.7° (Exps. 4 and 5). The information transfer of the PIP and MCP joint-angle position were similar, indicating 3--4 perfectly identifiable joint-angle positions for both joints (Exps. 3 and 6). The results provide the basic data needed for estimating, for example, the resolution of fingertip position during active free motion. They are compared to the results from previous studies on joint position, length, and thickness perception.

We report position and force detection thresholds for sinusoidal waveforms in the frequency range 10-500 Hz delivered through a stylus. The participants were required to hold the stylus in a way similar to that of holding the stylus of a force-feedback device. A minishaker moved the stylus along its length so that the majority of vibrations were presented tangentially to the skin of the hand. The measured position thresholds decreased initially with an increasing stimulus frequency and formed a U-shaped curve in the high frequency region. The thresholds of high frequency vibrations were lower than those reported previously for vibrations that were perpendicular to the skin, but were similar to the thresholds reported earlier using vibrations that were tangential to the skin. A similar force threshold curve was obtained using a force sensor attached to one end of the stylus. Mechanical impedance of the skin derived from velocity estimates and force measurements indicated that the skin and tissues in the hand holding the stylus can be modeled with mass-, damper- and spring-like elements. A comparison of the mechanical impedance from the present study with those reported previously showed similar results for vibrations delivered in the tangential and normal directions to the skin

Despite a wealth of literature on discrimination thresholds for displacement, force magnitude, stiffness, and viscosity, there is currently a lack of data on our ability to discriminate force directions. Such data are needed in designing haptic rendering algorithms where force direction, as well as force magnitude, are used to encode information such as surface topography. Given that haptic information is typically presented in addition to visual information in a data perceptualization system, it is also important to investigate the extent to which the congruency of visual information affects force-direction discrimination. In this article, the authors report an experiment on the discrimination threshold of force directions under the three display conditions of haptics alone (H), haptics plus congruent vision (HVcong), and haptics plus incongruent vision (HVincong). Average force-direction discrimination thresholds were found to be 18.4°, 25.6°, and 31.9° for the HVcong, H and HVincong conditions, respectively. The results show that the congruency of visual information significantly affected haptic discrimination of force directions, and that the force-direction discrimination thresholds did not seem to depend on the reference force direction. The implications of the results for designing haptic virtual environments, especially when the numbers of sensors and actuators in a haptic display do not match, are discussed.

The force-constancy hypothesis states that the user of a force-feedback device maintains a constant penetration force when stroking virtual surfaces in order to perceive their topography. The hypothesis was developed to address a real-world data perceptualization problem where the perception of surface topography was distorted when the surface stiffness was nonuniform. Two experiments were conducted. In Experiment I, we recorded the penetration depths of the probe tip while the user stroked two surfaces with equal height but different stiffness values. We found that the data could be quantitatively modeled by the force-constancy hypothesis when the virtual surfaces were neither too soft nor too hard. In Experiment II, we demonstrated that given two adjacent surfaces, their perceived height difference depended on both the surface stiffness values as well as the relative heights of the surfaces. Specifically, we showed that the higher but softer surface could be perceived to be lower, at the same height, or higher than the other surface, depending on how much higher it was than the other surface. The results were consistent with the predictions of the force-constancy hypothesis. Our findings underscore the importance of understanding the interplay of haptic rendering parameters.

This study investigates the effect of update rate on the perceived quality of haptic virtual textures, focusing on one type of perceived instability called "buzzing." Buzzing refers to the high-frequency noises that may emanate from a force-feedback device during haptic texture rendering. We first present a simulation of a haptic texture rendering system that explicitly models the sampled-data nature of the system. The simulation shows that a slow haptic update rate can significantly increase the high-frequency noise in the reconstructed force command sent to the haptic interface. This noise may excite the structural resonance of the haptic interface and result in a high-frequency buzzing at the interaction tool that a user holds. A subsequent psychophysical experiment, conducted over a wide range of update rate (250 Hz - 40 kHz), has confirmed the simulation results. The results show a 278% increase in the maximum stiffness (0.325 - 0.904 N/mm over the update rates tested) that could be used for rendering perceptually "clean" virtual haptic textures. These results argue for haptic update rates that are much higher than the widely-accepted value of 1 kHz. Considering that an application requiring hard surface rendering needs a stiffness value of at least 0.8 - 1.0 N/mm, we recommend a haptic update rate in the range 5-10 kHz for perceptually stable haptic texture rendering.

This paper reports new findings of our ongoing research on perceived instability that human users frequently experience from interacting with virtual textures rendered with a force-feedback haptic interface. Our work is aimed at a better understanding of a new type of perceived instability called "aliveness", which was discovered during our previous psychophysical experiments performed using a popular texture rendering method (spring model with fixed force directions) and a common texture model (position-based sinusoidal grating). We first examine the perceptual and physical characteristics of the proximal stimuli that cause the perception of aliveness in virtual textures. It leads to the hypothesis that the virtual environment model used for computing texture-perturbing forces, not the traditional control-related instabilities of the haptic texture rendering system, is responsible for aliveness perception. We show that this conjecture is true by applying passivity-based stability theory to the position and force data measured during user interaction with virtual textures. Examples of the data where the haptic texture rendering system is passive (therefore stable) and aliveness is perceived are provided to substantiate our conclusions. Our results point to the importance of designing haptic texture rendering methods and models that are free of perceptual artifacts.

Previous research has shown that the use of multimodal signals can lead to faster and more accurate responses compared to purely unimodal displays. However, in most cases response facilitation only occurs when the signals are presented in roughly the same spatial location. This would suggest a severe restriction on interface designers: to use multimodal displays effectively all signals must be presented from the same location on the display. We previously reported evidence that the use of haptic cues may provide a solution to this problem as haptic cues presented to a user's back can be used to redirect visual attention to locations on a screen in front of the user (Tan et al., 2001). In the present experiment we used a visual change detection task to investigate whether (i) this type of visual-haptic interaction is robust at low cue validity rates and (ii) similar effects occur for auditory cues. Valid haptic cues resulted in significantly faster change detection times even when they accurately indicated the location of the change on only 20% of the trials. Auditory cues had a much smaller effect on detection times at the high validity rate (80%) than haptic cues and did not significantly improve performance at the 20% validity rate. These results suggest that the use haptic attentional cues may be particularly effective in environments in which information cannot be presented in the same spatial location.

This paper reports on the results of our current research towards a qualitative characterization of the instability that a human user often perceives from a virtual textured surface rendered with a force-reflecting haptic interface. Based on our previous study of perceptual instability during haptic texture rendering, an experiment is designed and conducted in order to measure the proximal stimuli delivered by the virtual textures in terms of several physical variables (position, force, and acceleration). The measured data are analyzed in the frequency domain and characterized by sensation levels with regards to the human detection thresholds for temporal stimuli. In particular, a spectral band that conveys texture information and another one that induces instability perception are identified along with their associated sensation levels. These frequency bands excite different mechanoreceptors in the skin, and are therefore perceptually distinctive.