Social touch is a rich channel of human communication, conveying emotion, intent, and meaning embedded in context. Yet most HCI studies treat touch in isolation, overlooking the layered subtleties that shape interpretation. We present a contextual analysis of 5,016 social touch events, grounded in a large collection of annotated scenes from films, dramas, and documentaries. Using a computer vision pipeline, we segmented touch events from video and annotated them across dimensions, including who is involved, how the gesture is performed, where on the body it occurs, and the cultural backdrop. Our analysis shows that identical gestures can convey distinct meanings depending on body location, relationship type, and context. Similar intentions—like comfort, encouragement, or dominance—may be expressed through different gestures or locations, shaped by relational dynamics, cultural norms, and public or private settings. These insights inform the design of socially aware touch technologies, including avatars, social agents, and mediated communication systems.

Ring interfaces have gained attention in wearable technology for their lightweight and hands-free design. However, their compact form factor limits them to conveying simple information, such as direction or notification, through vibration, electrotactile, or force feedback. In this paper, we introduce HaRing, a novel haptic ring interface equipped with a 4 × 6 pin array display. This dynamic display delivers rich spatial patterns that simple vibration cannot express, effectively conveying high-dimensional information such as directions, semantic symbols, and letters. Its design enables one-handed, eyes-free interaction that does not interfere with visual tasks. We conducted a series of perceptual and user studies to demonstrate its effectiveness, showing a high recognition accuracy of over 94% for complex letters after a brief training period. We anticipate that HaRing can serve as an innovative haptic-only interface for multitasking in real-world or VR environments with high visual load.

Haptic feedback has become a common feature in game experiences, yet little is known about how its effects differ between active players and passive spectators. This study investigated how haptic feedback influences user experience and technology acceptance in the context of first-person shooter (FPS) games, particularly by comparing its effects on active players and passive spectators. An experiment with 60 participants tested four conditions defined by two factors: haptic feedback (present vs. absent) and user role (player vs. spectator). The results showed that haptic feedback enhanced the intention to use games in both roles, with a stronger effect among spectators. Players’ intention was primarily driven by perceived enjoyment through a hedonic pathway, whereas spectators responded to both perceived enjoyment and perceived usefulness through both hedonic and eudaimonic pathways. These findings highlight the need for role-sensitive haptic design and validate two pathways of the Haptic-Augmented Game Technology Acceptance Model (HAG-TAM) for gaming experiences.

This paper presents HaptiCraft, a modular handheld haptic controller designed to replicate the appearance, mass distribution, and multimodal haptic properties of virtual objects. HaptiCraft consists of many modules that provide distinct functions for assembly and multimodal haptic feedback, supporting five types: vibration, impact, thermal, variable inertia, and variable stiffness. Users can readily assemble these modules to create a controller tailored to their needs. To simplify the design process, especially for novice users, we also provide a graphical authoring tool and assess its usability. Finally, we evaluate the system's effectiveness through various virtual reality (VR) scenarios, demonstrating that HaptiCraft significantly enhances user experience. HaptiCraft makes a substantial contribution to the field of VR handheld controllers by offering comprehensive support for shape-changing, variable inertia, and rich multimodal haptic feedback, coupled with an effective authoring method.

Creating interactive VR content typically requires extensive modeling effort, making scalable production difficult. We present an automated computational pipeline that converts a single image into an interactive 3D scene with plausible kinesthetic feedback. Our approach leverages a Large Language Model (LLM) to detect objects in the image and infer haptic properties from visual and textual cues. These extracted results are then utilized to synthesize 3D models and optimize haptic properties for perceptual distinguishability, while combining them into a complete interactive environment. A user study shows that the generated VR scenes provide compelling visuo-haptic experiences, highlighting the potential of our method for scalable multisensory world generation. To our knowledge, this is the first system to automatically produce VR scenes with force feedback from a single image, pointing toward a practical direction for lowering the barriers to creating haptically enriched VR content.

Figure 1: Concept of our motion-tuning method.Our method is for modulating the drafts of motion effects automatically in the presence of video for the best quality of the user experience.It allows designers to easily create different versions of motion effects that can provide the designed user experiences, i.e., user feelings.Our method takes as input the draft motion effect and target adjective scores for perceiving both video and motion effects, as determined by the designers.It first converts the input scores into motion-only scores, leveraging the regression models addressing audiovisual effects, and then generates a new motion effect modulated for the converted scores from the draft.

Haptic motion effects refer to physical stimuli that stimulate the human vestibular system for various purposes, such as flight simulation and entertainment. Motion effects are intensively utilized in 4D films, which provide viewers with multisensory special effects with audiovisual stimuli. This paper focuses on the automatic creation of motion effects that express the physical movements in human dance. Such effects can significantly enhance the user’s multisensory experiences of watching and feeling a human dance. Taking the human motion capture data as input, our algorithms compute full six-degree-of-freedom motion effects featuring: (1) Separate processing of the external and internal movements of a dancer’s body; (2) Decomposing the internal movements into a few segments to obtain optimal motion effects between clarity and expressiveness; and (3) Merging of all external and internal movements with appropriate scaling and weights. Our novel method can elicit better user experiences than two previous motion effect generation algorithms, which respectively represent the movements in general articulated bodies and all dynamic components in the scene.

This study aims to identify and classify the factors influencing the emotional experience of audiences engaging with four-dimensional (4D) content. Through systematic literature reviews, expert workshops, and experiments, factors were identified and refined. Factors were first identified through a systematic literature review, then expanded with expert insights on 4D content design and user experience. Finally, audience experiences were analyzed through interviews following experiments with 32 participants, further refining the factors. The 151 collected factors were categorized based on common characteristics, resulting in the identification of 5 major groups and 25 subgroups. This classification system highlights the diversity of factors such as content experience, viewing environment, sensory stimulus design, personal characteristics, and content composition, emphasizing the importance of combining these elements to enhance emotional engagement. Furthermore, it provides a valuable theoretical foundation and practical guidance for 4D content creators, enabling them to design more immersive and emotionally resonant experiences.

The implementation of digital twins (DTs) in industrial environments poses significant challenges, especially for tasks requiring direct physical interaction. This is critical in high-risk environments where system failures can lead to severe human and economic losses, and where even minor errors can have catastrophic consequences. Traditional robotic teleoperation systems often lack intuitive control and require extensive training, increasing the risk of malfunction and accident. This study addresses these challenges by introducing DT-based teleoperation for steel production, focusing on the hazardous lump iron removal process. We present two novel haptic interfaces: POstick-KF, offering enhanced kinesthetic feedback, and POstick-VF, providing a larger workspace with visuo-tactile feedback. These interfaces are designed to mimic real tools, ensuring intuitive control and quick operator training. A user study shows that POstick-VF significantly improves tracking accuracy for novices, while POstick-KF excels in interaction performance regardless of experience. In particular, POstick-VF improves interaction skills over time, and achieves comparable performance to POstick-KF, but with a simpler system. These interfaces have been rigorously tested to ensure robustness and are currently undergoing field testing for real-world applications. Our findings highlight the potential of these haptic devices to achieve the safety and efficiency of DT-based teleoperation within high-risk industrial environments, marking a significant advance in the field.

Spatial audio and directional tactile cues are crucial for target localization and collision avoidance, especially when visual information is limited or cognitive load is high. As multisensory cues, spatially aligned auditory and tactile stimuli can perform better than misaligned ones, but achieving precise alignment is challenging. This study aims to (1) identify just noticeable differences (JNDs) of direction between auditory and tactile stimuli in the horizontal plane and (2) evaluate hazard avoidance performance with misaligned cues within the JNDs. The estimated JNDs increase from 26° to 84° as the stimuli shifted from the center to the side of the body. We also demonstrate that perceptual alignment of auditory and tactile cues within the JND ranges yielded comparable hazard avoidance performance and usability to exact physical alignment. Our study offers useful insights for more efficient and accurate spatial audio-tactile rendering systems for extended reality (XR).

In the realm of virtual reality (VR), shape-changing controllers have emerged as a means to enhance visuo-haptic congruence during user interactions. The major emphasis has been placed on manipulating the inertia tensor of a shape-changing controller to control the perceived shape. This paper delves deeper by exploring how the material properties of the controller’s handle, distinct from the inertial information, affect the perceived shape, focusing on the perceived length. We conducted three perceptual experiments to examine the effects of the handle’s softness, thermal conductivity, and texture, respectively. Results demonstrated that a softer handle increases the perceived length, whereas a handle with higher thermal conductivity reduces it. Texture, in the form of varying bumps, also alters the length perception. These results provide more comprehensive knowledge of the intricate relationship between perceived length and controller handle properties, expanding the design alternatives for shape-changing controllers for immersive VR experiences.

Motion effects are vital for enhancing 4D experiences in various applications, e.g., rides, films, and games, by physically moving the user. Recent research introduced algorithms for generating effective motion effects from the audiovisual stream, but these approaches either require object motion trajectories or miss important scene components, such as other objects, particles, and camera motion. This paper proposes a fully automatic algorithm synthesizing motion effects from all scene components. Our method leverages computer vision technologies to capture salient movements among multiple components, enabling the creation of holistic motion effects without the need for human intervention. Then, our method computes a motion proxy representing the movements of the scene components by compressing the captured movements into a singlepoint motion. The motion proxy is converted into a motion command through a motion cueing algorithm and delivered to the user. The results of a user study show that our algorithm can generate compelling motion effects that enhance the 4D experience better than semi-manually authored effects. Our approach can facilitate the production of captivating motion effects for many applications.

Touch is crucial in communicating different social intents in our everyday lives. However, we lack methods to bring the capability of touch to express various social signals into VR. This paper aims to bridge this gap by exploring the effectiveness of haptic feedback in conveying the social intent of a touch initiator (a user touching a virtual agent). In User Study 1, we observe the touch gestures that users employ to express different social intents and collect the haptic feedback parameters appropriate for representing these touch gestures. User Study 2 analyzes how social intent-dependent multimodal (pressure, thermal, and texture) haptic feedback affects the touch initiator’s virtual social experience. The results indicate that haptic feedback representing social intent strengthens the social expression of touch and fosters emotional closeness with virtual agents. Combining it with a physical proxy makes the effects even more effective, enhancing copresence and avatar embodiment.

This paper explores the impact of vibrotactile haptic feedback on heart activity when the feedback is provided at four different body locations (chest, wrist, neck, and ankle) and with two feedback rates (50 bpm and 110 bpm). A user study found that the neck placement resulted in higher heart rates and lower heart rate variability, and higher frequencies correlated with increased heart rates and decreased heart rate variability. The chest was preferred in self-reported metrics, and neck placement was perceived as less satisfying, harmonious, and immersive. This research contributes to understanding the interplay between psychological experiences and physiological responses when using haptic biofeedback resembling real body signals.

Motion effects are indispensable for improving 4D experiences in highly interactive applications, such as amusement parks, 4D theaters, and virtual reality games. Their recent emergence calls for effective algorithms generating motion effects synchronized with audiovisual content. This paper presents an automatic algorithm for synthesizing the object-based motion effects that express the movements of multiple articulated bodies inclusively when the objects’ motion trajectories are available in the 3D camera space. By taking the visual velocities and sizes of all object parts, our method computes a motion proxy that represents the objects’ movements by one point and converts the motion proxy to a motion command through a motion cueing algorithm. The motion proxy is determined by linearly combining the velocities, and its best combination was selected from several candidates by user studies. The results of user studies indicate that our algorithm can produce compelling object-based motion effects that enhance the multisensory experience.

We present a visuo-haptic crossmodal model of shape perception designed for shape-changing handheld controllers. The model uses the inertia tensor of an object to bridge the two senses. The model was constructed from the results of three perceptual experiments. In the first two experiments, we validate that the primary moment and product of inertia (MOI and POI) in the inertia tensor have critical effects on the haptic perception of object length and asymmetry. Then, we estimate a haptic-to-visual shape matching model using MOI and POI as two link variables from the results of the third experiment for crossmodal magnitude production. Finally, we validate in a summative user study that the inverse of the shape matching model is effective for pairing a perceptually-congruent haptic object from a virtual object—the functionality we need for shape-changing handheld interfaces to afford perceptually-fulfilling sensory experiences in virtual reality.

We propose an algorithm that generates a vibration, an impact, or a vibration+impact haptic effect by processing a sound signal in real time. Our algorithm is selective in that it matches the most appropriate type of haptic effects to the sound using a machine-learning classifier (random forest) that is built on expert-labeled datasets. Our algorithm is tailored to enhance user experiences for video game play, and we present two examples for the RPG (role-playing game) and FPS (first-person shooter) genres. We demonstrate the effectiveness of our algorithm by a user study in comparison to other state-of-the-art (SOTA) methods for the same cross-modal conversion. Our system elicits better multisensory user experiences than the SOTA algorithms for both game genres.

(Haptic) motion effects refer to the vestibular stimuli generated by a motion platform and delivered to the whole body of a user sitting on the platform. Motion effects are an essential tool for creating vivid sensory experiences in various extended reality (XR) applications, ranging from training simulators to recent 4D rides, films, and games for entertainment. For the latter purpose, motion effects emphasize audiovisual events occurring in the scene, such as camera motion, the movement of an object of interest, and special sounds. Recent research developed several algorithms to produce motion effects from the audiovisual stream automatically. However, these algorithms are designed for a single class of motion effects, and extension to multiple motion effect classes remains unexplored. In this paper, we propose an algorithmic framework that merges camera and object motion effects into one motion effect while preserving the perceptual consequences of the two effects. We validate the framework’s perceptual performance through a user study. To our knowledge, this work is one of the first successful reports of merging different kinds of motion effects for improved XR experiences.

One can embed a vibration actuator to a physical button and augment the physical button’s original kinesthetic response with a programmable vibration generated by the actuator. Such vibration-augmented buttons inherit the advantages of both physical and virtual buttons. This paper reports the information transmission capacity of vibration-augmented buttons. It was obtained by conducting a series of absolute identification experiments while increasing the number of augmented buttons. The information transmission capacity found was 2.6 bits, and vibration-augmented and physical buttons showed similar abilities in rendering easily recognizable haptic responses. In addition, we showcase a VR text entry application that utilizes vibration-augmented buttons. Our method provides several error messages to the user during text entry using a VR controller that includes an augmented button. We validate that the variable haptic feedback improves task performance, cognitive workload, and user experience for a transcription task.

In recent times, millions of people enjoy watching video gameplays at an eSports stadium or home. We seek a method that improves gameplay spectator or viewer experiences by presenting multisensory stimuli. Using a motion chair, we provide the motion effects automatically generated from the audiovisual stream to the viewers watching a first-person shooter (FPS) gameplay. The motion effects express the game character’s movement and gunfire action. We describe algorithms for the computation of such motion effects developed using computer vision techniques and deep learning. By a user study, we demonstrate that our method of providing motion effects significantly improves the viewing experiences of FPS gameplay. The contributions of this paper are with the motion synthesis algorithms integrated for FPS games and the empirical evidence for the benefits of experiencing multisensory gameplays.

As computing paradigms shift toward mobile and ubiquitous interaction, there is an increasing demand for wearable interfaces supporting multifaceted input in smart living environments. In this regard, we introduce a system that identifies contact fingers using vibration as a modality of communication. We investigate the vibration characteristics of the communication channels involved and simulate the transmission of vibration sequences. In the simulation, we test and refine modulation and demodulation methods to design vibratory communication protocols that are robust to environmental noises and can detect multiple simultaneous contact fingers. As a result, we encode an on-off keying sequence with a unique carrier frequency to each finger and demodulate the sequences by applying cross-correlation. We verify the communication protocols in two environments, laboratory and cafe, where the resulting highest accuracy was 93 % and 90.5 %, respectively. Our system achieves over 91 % accuracy in identifying seven contact states from three fingers while wearing only two actuator rings with the aid of a touch screen. Our findings shed light on diversifying touch interactions on rigid surfaces by means of vibratory communication.

Four-dimensional (4D) films, which provide special physical effects to the audience with audiovisual stimuli, are gaining more popularity and acceptance. One of the most frequent 4D effects is the object-based motion effect, which refers to the vestibular stimulus generated by a motion chair to emphasize a moving object of interest, e.g., the flying iron man, displayed on the screen. In this paper, we present an algorithm for synthesizing convincing object-based motion effects automatically from a given object motion trajectory. While previous approaches use the 2D object position on the screen as input, our method takes the 3D position and orientation of the object in the camera space and computes its motion proxy that reflects both the object translation and rotation, as well as its size to the viewers' eyes. The proxy is determined based on the results of a perceptual experiment that presents an optimal additive rule of the translation and rotation information scaled by the object's visual size. The motion proxy is fed to a motion cueing algorithm (MCA) that computes the command using a washout filter or model predictive control. The most appropriate MCA for our purpose is selected from six candidates by a user study. We also consider the effects of visual perception by incorporating two types of motion field equations into the computation of the visually perceived velocity. The results of a user study indicate that our algorithm can generate compelling object-based motion effects that better enhance the 4D film viewing experience than the previous methods.

In this paper, we report both absolute and differential thresholds for motion in the six cardinal directions as comprehensively as possible. As with general 4D motion effects, we used sinusoidal motions with low intensity and large frequency as stimuli. Hence, we could also compare the effectiveness of motion types in delivering motion effects. We found that the thresholds for the z-axis (up-down) were higher than those for the x-axis (front-back) and y-axis (left-right) in both kinds of thresholds and that the type of motion significantly affected both thresholds. Further, between differential thresholds and reference intensities, we found a strong linear relationship for roll, yaw and, surge. Compared to them, a relatively weak linear relationship was observed for the rest of the motion types. Our results can be useful for generating motion effects for 4D contents while considering the human sensitivity to motion feedback.

A motion platform provides the vestibular stimuli that elicit the sensations of self-motion and thereby improves the immersiveness. A representative example is a 4D Ride, which presents a video of POV shots and motion effects synchronized with the camera motion in the video. Previous research efforts resulted in a few automatic motion effect synthesis algorithms for POV shots. Although effective in generating gross motion effects, they do not consider fine features on the ground, such as a rough or bumpy road. In this paper, we propose an algorithm for styling the gross motion effects using a texture image. Our algorithm transforms a texture image into a high-frequency style motion and merges it with the original motion while respecting both perceptual and device constraints. A user study demonstrated that texture styling could increase immersiveness, realism, and harmony.

This article addresses a data-driven framework, modeling expert driving skills for performance-based haptic assistance using neural networks (NNs). We have built a haptic driving training simulator to collect expert driving data and to provide proper haptic feedback. We establish an expert driving skill model by training NNs with the collected data. Then, the skill model is applied to the performance-based haptic assistance to provide optimized references of the steering/pedaling movements. We evaluate the skill model and its application to the performance-based haptic assistance in two user experiments. The results of the first experiment demonstrate that our skill model has appropriately captured experts' steering/pedaling skills. The results of the second experiment show that our performance-based haptic assistance can help novice drivers perform steering as expert drivers, but cannot assist their pedaling performance.

In tactile interaction, a phantom sensation refers to an illusion felt on the skin between two distant points at which vibrations are applied. It can improve the perceptual spatial resolution of tactile stimulation with a few tactors. All phantom sensations reported in the literature act on the skin or out of the body, but no such reports exist for those eliciting sensations penetrating the body. This paper addresses tactile phantom sensations in which two vibration actuators on the dorsal and palmar sides of the hand present an illusion of vibration passing through the hand. We also demonstrate similar tactile illusions for the torso. For optimal design, we conducted user studies while varying vibration frequency, envelope function, stimulus duration, and penetrating direction. Based on the results, we present design guidelines on penetrating phantom sensations for its use in immersive virtual reality applications.

Physical buttons provide clear haptic feedback when pressed and released, but their responses are unvarying. Physical buttons can be powered by force actuators to produce unlimited click sensations, but the cost is substantial. An alternative can be augmenting physical buttons with simple and inexpensive vibration actuators. When pushed, an augmented button generates a vibration overlayed on the button's original kinesthetic response, under the general framework of haptic augmented reality. We explore the design space of augmented buttons while changing vibration frequency, amplitude, duration, and envelope. We then visualize the perceptual structure of augmented buttons by estimating a perceptual space for 7 physical buttons and 40 augmented buttons. Their sensations are also assessed against adjectives, and results are mapped into the perceptual space to identify meaningful perceptual dimensions. Our results contribute to understanding the benefits and limitations of programmable vibration-augmented physical buttons with emphasis on their feels.

This is a survey article on the information transmission capability of haptic devices and ways of maximizing it. It is intended for readers who are engineering professionals interested in developing novel haptic interfaces for a variety of applications but are not necessarily trained in haptic science or human user research. We posit that the ultimate goal of any interface is the exchange of information between a machine and a user, and as such, the evaluation should involve estimating the information-transmission capability with human users. We conducted a literature survey on studies of haptic devices evaluated with human users using an information theoretic framework. Our goal was to discover and summarize best practices that can lead to high information transmission. The results confirmed findings from our own previous studies, uncovered new ways to effectively increase information transmission, and pointed to the need for broader dissemination of proper experimental methodology. We, therefore, present a concise yet comprehensive tutorial on psychophysical methodology for estimating information transfer (IT) and IT rate with humans, survey results on the typical IT achievable with haptic devices, and guidelines for maximizing information transmission with any human-machine interfaces. Although we focus on haptic systems, the information-theoretic framework, the psychophysical methods, and the guidelines presented in this article are applicable to other sensory modalities and multimodal interface systems also.

We present VibEye: a vibration-mediated recognition system of objects for tangible interaction. A user holds an object between two fingers wearing VibEye. VibEye triggers a vibration from one finger, and the vibration that has propagated through the object is sensed at the other finger. This vibration includes information about the object's identity, and we represent it using a spectrogram. Collecting the spectrograms of many objects, we formulate the object recognition problem to a classical classification problem among the images. This simple method, when tested with 20 users, shows 92.5% accuracy for 16 objects of the same shape with various materials. This material-based classifier is also extended to the recognition of everyday objects. Lastly, we demonstrate several tangible applications where VibEye provides the needed functionality while enhancing user experiences. VibEye is particularly effective for recognizing objects made of different materials, which is difficult to distinguish by other means such as light and sound.

This paper presents TouchPhoto, which provides visual-audio-tactile assistive features to enable visually-impaired users to take and understand photographs independently. A user can take photographs under auditory guidance and record audio tags to aid later recall of the photographs’ contents. For comprehension, the user can listen to audio tags embedded in a photograph while touching salient features, e.g., human faces, using an electrovibration display. We conducted two user studies with visually-impaired users, one for picture taking and the other for understanding and recall, in a two-month interval. They considered auditory assistance as very useful for taking and understanding photographs and tactile features as helpful but to a limited extent.

We propose a data-driven method for realistic texture rendering on an electrovibration display. To compensate the nonlinear dynamics of an electrovibration display, we use nonlinear autoregressive with external input (NARX) neural networks as an inverse dynamics model of an electrovibration display. The neural networks are trained with lateral forces resulting from actuating the display with a pseudo-random binary signal (PRBS). The lateral forces collected from the textured surface with various scanning velocities and normal forces are fed into the neural network to generate the actuation signal for the display. For arbitrary scanning velocity and normal force, we apply the two-step interpolation scheme between the closest neighbors in the velocity-force grid.

Displacement information is essential in many robotic applications involving contact, especially with viscoelastic objects. In this paper, we provide a novel displacement estimation method using contact pressure distribution. Our method can estimate displacement using only a typical tactile sensor, which can be shared by a force measurement. As a result, our method can reduce the number of required sensors for stiffness estimation, so it is suitable for stiffness-required applications with strict spatial constraints. Simulation and experimental results demonstrate the performance of our method.

A phantom sensation refers to an illusory tactile sensation perceived midway between multiple distant stimulations on the skin. Phantom sensations have been used intensively in tactile interfaces owing to their simplicity and effectiveness. Despite that, the perceptual performance of phantom sensations is not completely understood, especially for 2D cases. This work is concerned with 2D stationary phantom sensations and their fundamental value as a means for information display. In User Study 1, we quantified the information transmission capacity using an absolute identification task of 2D phantom sensations. In User Study 2, we probed the distributions of the actual perceived positions of 2D phantom sensations. The investigations included both types of phantom sensations-within and out of the body. Our results provide general guidelines as to leveraging 2D phantom sensations in the design of spatial tactile display.

In this paper, we present two methods to substitute motion effects using vibrotactile effects in order to improve the 4D experiences of viewers. This work was motivated by the needs of more affordable 4D systems for individual users. Our sensory substitution algorithms convert motion commands to vibrotactile commands to a grid display that uses multiple actuators. While one method is based on the fundamental principle of vestibular feedback, the other method makes use of intuitive visually-based mapping from motion to vibrotactile stimulation. We carried out a user study and could confirm the effectiveness of our substitution methods in improving 4D experiences. To our knowledge, this is the first study that investigated the feasibility of replacing motion effects using much simpler and less expensive vibrotactile effects.

This paper presents a method that automatically transforms a virtual environment (VE) according to the mood of input music. We use machine learning to extract a mood from the music. We then select images exhibiting the mood and transfer their styles to the textures of objects in the VE photorealistically or artistically. Our user study results indicate that our method is effective in transferring valence-related aspects, but not arousal-related ones. Our method can still provide novel experiences in virtual reality and speed up the production of VEs by automating its procedure.

In haptics, two major modalities, force and vibration, are used to model real textures and recreate them in a virtual environment. This paper compares the perceptual advantages and disadvantages between the two approaches by a user study. In particular, the perceptual similarity of a virtual texture to a real texture is rated using five criteria of geometry, roughness, hardness, friction and overall similarity. These categorical comparisons allowed us to provide general guidelines to appropriate uses of the two approaches.

Haptic feedback has been finding increasing uses in affective interaction design. In this paper, we describe the emotional characteristics of multimodal tactile stimuli that combine vibrotactile and thermal stimuli. While varying three vibrotactile parameters of frequency, amplitude, and duration and one thermal parameter of constant temperature, we experimentally evaluated the valence and arousal of 1) thermal stimuli and 2) multimodal vibrotactile-thermal stimuli with 20 participants. Results indicated that the thermal and vibrotactile parameters have clear and somewhat independent effects on emotional responses. The findings of this study can contribute to expanding the design space of affective user interaction involving touch.

In this paper, we present a virtual reality based content/system called the "Drop the beat" designed to help the mindfulness and train one to overcome panic disorder. The two main elements of the proposed system are the (1) use of 360-degree video for presenting the panic inducing situation and, (2) facilitation of the mindfulness through a compelling scenario and immersive experience with multimodal feedback. In particular, we hypothesized that the direct observance and tangibly feeling for the beating heart in one's hand would help the user train to rationalize and overcome the situation (and e.g. effectively bring down one's heart rate back to a normal level). We conducted a small pilot study, administering the proposed VR content to five panic disorder patients and report the interim results.

This paper presents an algorithm that allows for the rapid design of motion effects for 4D films. Our algorithm is based on a viewer-centered rendering strategy that matches chair motion to the movement of a viewer's visual attention. Object tracking algorithm is used to estimate the movement of visual attention under the assumption that visual attention follows an object of interest. We performed several experiments to find optimal parameters for implementation, such as the required accuracy of object tracking. Our algorithm enables motion effects design to be at least 10 times faster than the current practice of manual authoring. We also assessed the subjective quality of the motion effects generated by our algorithm, and results indicated that our algorithm can provide perceptually plausible motion effects.

4D film is an immersive entertainment system that presents various physical effects with a film in order to enhance viewers' experiences. Despite the recent emergence of 4D theaters, production of 4D effects relies on manual authoring. In this paper, we present algorithms that synthesize three classes of motion effects from the audiovisual content of a film. The first class of motion effects is those responding to fast camera motion to enhance the immersiveness of point-of-view shots, delivering fast and dynamic vestibular feedback. The second class moves viewers as closely as possible to the trajectory of slowly moving camera. Such motion provides an illusional effect of observing the scene from a distance while moving slowly within the scene. For these two classes, our algorithms compute the relative camera motion and then map it to a motion command to the 4D chair using appropriate motion mapping algorithms. The last class is for special effects, such as explosions, and our algorithm uses sound for the synthesis of impulses and vibrations. We assessed the subjective quality of our algorithms by user experiments, and results indicated that our algorithms can provide compelling motion effects.

This paper reports our recent finding that a laser that is radiated on a thin light-absorbing elastic medium attached on the skin can elicit a tactile sensation of mechanical tap. Laser radiation to the elastic medium creates inner elastic waves on the basis of thermoelastic effects, which subsequently move the medium and stimulate the skin. We characterize the associated stimulus by measuring its physical properties. In addition, the perceptual identity of the stimulus is confirmed by comparing it to mechanical and electrical stimuli by means of perceptual spaces. All evidence claims that indirect laser radiation conveys a sensation of short mechanical tap with little individual difference. To the best of our knowledge, this is the first study that discovers the possibility of using indirect laser radiation for mid-air tactile rendering.

The rapid development of nano technology and mobile devices has created high demand for shape changing mobile devices in order to provide users with more functions in less space and improved usability. However, the fundamental incompatibility between flexible displays and rigid actuator components has brought a huge barrier for commercialization of this technology. This paper presents a flexible and bendable vibrotactile actuator that can be easily applied to shape changing mobile devices. The proposed vibrotactile actuator is made with an electro-conductive membrane based on polyurethane, a base membrane, and an airgap. Actuation of the proposed actuator is controlled by a polarity of both charged membranes and the actuator performance can be modulated by increasing level of biased electric potential. A user experiment showed that the proposed actuator can provide vibrations with sufficient strength for perception.

Vibrotactile flows refer to vibrotactile sensations that move continuously on the surface of a mobile device. They have been studied to render one-dimensional illusory movements using two actuators. In this work, we extend the dimension of vibrotactile flows to two dimensions by means of edge flows-vibrotactile flows that rotate along the edges of a rectangular mobile device, rendered using four actuators placed at its four corners. We designed 32 edge flows and carried out a longitudinal user study in order to measure the information transmission capacity of edge flows, while looking into the effects of practice and correct-answer feedback on the identification accuracy of edge flows. Results showed that the information transmission capacity of the edge flows was 3.70 bits, which is greater than for most previous studies; that some extent of practice is required for robust identification, but it can take place quickly; and that correct-answer feedback is advantageous but its degree of improvement was less than that of repeated practice. Our findings can contribute to enlarging the design space of tactile stimuli for robust transmission of abstract information to mobile device users.

This paper presents a new approach to data-driven modeling of isotropic haptic textures using frequency-decomposed neural networks from the contact acceleration data that are captured when a stylus is scanned on a textured surface with diverse scanning velocities and normal forces. We first describe a motorized texture scanner that was developed for accurate and easy data collection under a wide variety of conditions. We then propose two neural network models with different topologies: a unified model that feeds all of acceleration data, scanning velocity, and normal force as input variables to a single large neural network and a decomposed model that consists of a number of smaller neural networks each of which is trained with the acceleration data for a pair of scanning velocity and normal force. An experiment with real samples showed that the unified model has better cross-validation ability in terms of spectral rms errors and its performance is comparable to the best available in the literature. We also present some preliminary results of anisotropic texture modeling achieved by extending the unified model.

This paper presents an extended data-driven haptic rendering method capable of reproducing force responses during sliding interaction on a large surface area. The core part of the approach is a novel input variable set for data interpolation model training, which includes the position of a proxy - the contact point on a surface when undeformed. This enables us to estimate friction in both sliding and sticking states. The behavior of the proxy is simulated in real-time based on a sliding yield surface - a cone-shaped surface separating the sliding and the sticking area in the external force space, identified through an automated palpation procedure. During modeling, input data and resultant force data are collected through manual palpation of a target object, which are used to build a radial-basis interpolation model. During rendering, this interpolation model with interaction inputs estimates force responses in real-time. Performance evaluation shows that our approach achieves reasonable estimation accuracy. This work is among the first attempts that fully support sliding exploration in the data-driven paradigm.

This paper is concerned with emotional responses of tactile icons. Using three sets of tactile icons in which four physical parameters-amplitude, frequency, duration, and envelope-were systematically varied, we estimated their valence and arousal scores in a perceptual experiment with 24 participants. Results showed that the four parameters have clear relationships to the emotional responses of tactile icons. Our tactile icons spanned to a large region in the valence-arousal space, but they did not elicit very positive-relaxing or very negative-relaxing emotional responses. These findings provide the design guidelines of tactile icons that have desired emotional properties.

Perceptual quality for media compression algorithms is traditionally evaluated through user studies. Such studies are time consuming, laborious and expensive, slowing down the development of new signal processing algorithms. To address this problem, a number of algorithmic quality prediction methodologies have been developed in the audio and video fields, something that is currently lacking in haptics research. In this paper, we present a novel method for predicting the perceptual quality degradation of compressed haptic texture signals. For this purpose, abstract perceptual features like Roughness, Brightness, etc. that capture the subjective experience of textures are exploited, in addition to low-level psychophysical models from the literature. As compared to the state-of-the-art, the presented prediction methodology shows an approximately 30% improvement in explaining the variance in the perceptual data.

This paper presents a novel indicator for lump detection that uses only contact pressure distribution, as well as its feasibility evaluation. The conventional lump detection methods are based on a pressure map obtained from contact pressure distributions, but they requires an adequate regulation of the contact condition during measurements. An alternative approach is to use a stiffness map, for which displacement measurement is necessary using additional devices or precise motion control. The proposed indicator relies on only the difference between the normal force and the surface normal obtained from contact pressure distributions. Thus, contact condition or probe motion does not affect the performance of the proposed method. The feasibility of the proposed method was evaluated by simulations and experiments, both of which showed promising results.

Tactile icons, brief tactile stimuli conveying abstract information, have found their use in various applications, and their use with visual elements is increasing on touchscreen user interfaces. However, effective design guidelines of tactile icons for crossmodal use have not been established. This paper addresses this problem by investigating the congruence between visual and tactile icons based on the hypothesis that emotional agreement between the icons improves congruence. The validity of this hypothesis was examined in three experiments. In Exp. I, we selected common visual icons and estimated their emotional responses using the circumplex model of affect. Tactile icons to be used as a pair were designed in Exp. II by varying their amplitude, frequency, and envelope (rhythm). Their emotional responses were also evaluated. In Exp. III, the congruence of 192 crossmodal icons made by combining the visual icons (8) and the tactile icons (24) was evaluated, and these congruence scores were compared with the valence and arousal scores of the two unimodal icons obtained in Exp. I and II. Experimental results suggested that the congruence of a crossmodal icon highly depends on the agreement in the emotional responses between its visual and tactile icons. This finding provides feasibility to the development of general design guidelines and heuristics for crossmodal icons that rely on the relationship between the emotional responses from the individual modalities. Our approach is expected to advance the current practice that associates the physical parameters between the different senses with better intuitiveness and simplicity.

This paper proposes a contact force decomposition method using tactile information for haptic augmented reality (AR). Haptic AR modulates real haptic interaction to desired interaction by adding virtual haptic feedback. For haptic AR rendering, the measurement of real haptic interaction including deformation and friction force is important for accurate rendering, but the current haptic AR algorithms do not support it properly. The proposed method, derived from a simple model of rigid-deformable body contact, decomposes contact force into deformation force and friction force using force and tactile information. A finite element method (FEM) based simulation of contact between two objects was performed for the performance verification of the proposed method. The results showed the proposed method can decompose contact force more exactly than the conventional decomposition methods in haptic AR.

In this paper, we propose a real-time perception-level audio-to-vibrotactile translation algorithm. Unlike previous signal-level conversion methods, our algorithm considers only perceptual characteristics, such as loudness and roughness, of audio and tactile stimuli. This perception-level approach allows for designing intuitive and explicit conversion models with clear understandings of their perceptual consequences. Our current implementation is tailored to accurate detection of special sound effects to provide well-synchronized audio-tactile feedback in immersive applications. We also assessed the performance of our translation algorithm in terms of the detection rate of special sound effects, computational performance, and user preference. All the experimental results supported that our algorithm works well as intended with better performance than the signal-level conversion methods, especially for games. Our algorithm can be easily realized in current products, including mobile devices, gaming devices, and 4D home theater systems, for richer user experience.

We introduce a learning system for the sight reading skill of simple drum sequences. Sight reading is a cognitive-motor skill that requires reading of music symbols and actions of multiple limbs for playing the music. The system provides knowledge of results (KR) pertaining to the learner's performance by color-coding music symbols, and guides the learner by indicating the corresponding action for a given music symbol using additional auditory or vibrotactile cues. To evaluate the effects of KR and guidance cues, three learning methods were experimentally compared: KR only, KR with auditory cues, and KR with vibrotactile cues. The task was to play random 16-note-long drum sequence displayed on a screen. Thirty university students learned the task using one of the learning methods in a between-subjects design. The experimental results did not show statistically significant differences between the methods in terms of task accuracy and completion time. This suggests that KR can be dominant for learning the task and the role of guidance cues can be subsidiary.

The objective of the current work is a driving training system to capture and transfer skills from the expert to the novice drivers. The emphasis is on the haptic feedback applied on the steering wheel and the accelerator pedal to improve trainee's performance. For capturing skills, driving signals of an expert driver, performing a given task using a developed driving simulator, are recorded as reference trajectories. For transferring, two different methods are considered: guidance to assist and disturbance to distract following the reference. For evaluation, two performance measures are assumed: trajectory-based and model-based. They are used not only to evaluate the progress of the trainee but also to tune the gain of controllers delivering haptic feedback.

We propose a vibrotactile pattern authoring method based on the user's demonstration, where a device generates a vibrotactile pattern by imitating the touch input pattern of the user. The usability of our demonstration-based method is also compared with the conventional waveform-based authoring. The results showed that our demonstration-based authoring delivered comparable or superior performance to the waveform-based method, depending on the specific design tasks.

This paper reviews the technology and applications of vibrotactile display, an effective information transfer modality for the emerging area of haptic media. Our emphasis is on summarizing foundational knowledge in this area and providing implementation guidelines for application designers who do not yet have a background in haptics. Specifically, we explain the relevant human vibrotactile perceptual capabilities, detail the main types of commercial vibrotactile actuators, and describe how to build both monolithic and localized vibrotactile displays. We then identify exemplary vibrotactile display systems in application areas ranging from the presentation of physical object properties to broadcasting vibrotactile media content.

This paper investigates the effect of haptic feedback on the learning of a 2-D sequential selection task, used as an abstraction of complex industrial manual assembly tasks. This mnemonic-motor task requires memorization of the selection order of points scattered on a 2-D plane and reproduction of this order using entire arm movements. Four information presentation methods, visual information only, visual information + enactment, visual information + haptic guidance, and visual information + haptic disturbance, are considered. The latter three methods provide different levels of haptic kinesthetic feedback to the trainee. We carried out a user study to assess the quantitative performance differences of the four training methods using a custom-built visuo-haptic training system. Experimental results showed the relative advantages and disadvantages of each information presentation method for both short-term and long-term memorization. In particular, training with only visual information was the best option for short-term memory, while training also with haptic disturbance was the most effective for long-term memory. Our findings have implications to designing a training method that is suitable for given training requirements.

Vibrotactile signals convey the touch-based characteristics of object surfaces felt through a tool. They particularly enhance the quality of human-machine interactions by providing realistic haptic perception of textures. In this paper, inspired by the similarities observed between vibrotactile texture signals and speech signals, we present a novel vibrotactile texture codec for bilateral teleoperation, based on well-known speech coding techniques. The proposed low bitrate, high quality codec preserves not only the spectral signature vital to the general feel of the texture, but also important temporal features of the texture signal. We report a compression ratio of 8:1 (12.5 %) with a constant output bitrate of 4 kbps, and we validate the perceptual transparency of the codec via rigorous subjective tests and analyses.

A haptic icon is a short physical stimulus attached to a simple meaning, which provides information and feedback to a user. To scale the utility demonstrated for small icon sets to larger ones, we need efficient strategies to help users learn subtle distinctions among stimuli, in a modality for which they may not hold detailed descriptive percepts. This paper investigates the effect of naming haptic stimuli - i.e. explicitly creating a linguistic marker - on the accuracy with which users are able to identify, distinguish, and recall stimuli. We conducted a between-subjects experiment using 60 participants equally divided among three naming conditions: no names, pre-selected non-descriptive names, and self-selected names. The experiment examined the impact of naming strategy on the ability of participants to identify stimuli in a nonverbal matching test, and on remembering stimulus names. For this challenging task and the degree of learning afforded, naming did not significantly impact accuracy of matching stimuli to meanings for all participants. However, more than twice of many of those allowed to choose names reported the ability to remember and distinguish stimuli than those required to use non-descriptive names, and many participants felt that the names were useful. Of middle-performing participants, the self-selected names group performed significantly better than the non-descriptive names group, and appeared to progress more quickly in learning. We summarize evidence for a trend that might widen with refined naming strategies and more extensive learning.

Haptic augmented reality merges real and virtual feedback, allowing a user to touch the real environment augmented with virtual haptic stimuli. A key functionality of this technology is altering the haptic properties of real objects by means of virtual haptic feedback. Previously, we have developed a haptic augmentation system in which object stiffness was modulated. In this paper, we extend our framework to cover further haptic properties: friction and weight. Simple but effective algorithms for estimating and altering these properties have been developed. The first approach allows us to change the inherent friction between a tool tip and a surface to a desired one identified in an offline process. The second technique enables a user to perceive an altered weight when lifting an object at two interaction points. The performance of the proposed algorithms has been evaluated on real objects. The errors remain within reasonable bounds and compare well to the relevant perceptual Weber fractions. Limitations of our technique are identified, and possible extensions are proposed.

This paper proposes a new interactive interface TAXEL, which aims at developing a reconfigurable self-morphing visio-haptic interface. We first present the overall architecture and concept of a self-morphing visuo-haptic interface. Key hardware components are developed and tested, including three tactile actuators using a piezoelectric active linear actuator, a passive MR fluid actuator, and a thin film-type actuator, respectively, and a flexible visual display based on the light-waveguide technology. Using the developed components, a tactile platform that includes a 8×16 array of the linear actuators is implemented for a proof of concept. A rendering engine is also designed for the tactile platform with emphasis on the use of haptic feedback together with GUI. We also carried out a user study with virtual button simulation as a benchmark to evaluate the performance of the TAXEL tactile platform. Lastly, an integrated system with a visual display is demonstrated along with several application examples.

Amplitude-modulated vibrotactile stimuli have been broadly used for tactile perception and rendering research. However, understandings of the perceptual relations between amplitude-modulated vibrations are not comprehensive yet. In this paper, we present a perceptual analysis on the identifying characteristics of amplitude-modulated vibrations and their perceptual relations. We estimated the perceptual dissimilarities among one carrier and seven amplitude-modulated sinusoidal vibrations (150-Hz carrier frequency; seven modulation frequencies in 1-80 Hz) in a psychophysical experiment. The dissimilarity scores were inspected using multi-dimensional scaling to obtain an appropriate perceptual space. The optimal perceptual space was two-dimensional, where the vibration points exhibited a circular formation. The analysis showed that the pulse-like low-frequency sensation of an amplitude-modulated vibration increased for very low modulation frequencies (1-10 Hz), then decreased for higher modulation frequencies (10-80 Hz), and eventually converged to the smooth vibrational sensation of the 150-Hz carrier signal. It also suggested that the envelope waveform is primary information for the discrimination of amplitude-modulated signals, instead of their spectral energy distributions. These findings can contribute to the design of perceptually salient and distinctive vibrotactile signals using amplitude modulation.

In this paper, we present the design of a large number (72) of tactile stimuli for the confirmation feedback of virtual button presses on the mobile device touchscreen. Two industry standard variable-reluctance actuators (enhanced voice-coil actuators with added mass for stronger output) were used in the study. The design parameters of an input were amplitude, duration, carrier signal, envelope function, and actuator. Eighteen participants evaluated the modeled patterns with the criteria of similarity to physical buttons and user preference. An adjective rating task was also accompanied to assess the subjective quality of the designed tactile effects. Experimental results unveiled several important guidelines for designing realistic and favorable button-click tactile feedback. The findings of this paper have implications for improving the usability of user interface components displayed on a touchscreen by means of haptic feedback.

This article evaluates the usability of motion sensing-based interaction on a mobile platform using image browsing as a representative task. Three types of interfaces, a physical button interface, a motion-sensing interface using a high-precision commercial 3D motion tracker, and a motion-sensing interface using an in-house low-cost 3D motion tracker, are compared in terms of task performance and subjective preference. Participants were provided with prolonged training over 20 days, in order to compensate for the participants’ unfamiliarity with the motion-sensing interfaces. Experimental results showed that the participants’ task performance and subjective preference for the two motion-sensing interfaces were initially low, but they rapidly improved with training and soon approached the level of the button interface. Furthermore, a recall test, which was conducted 4 weeks later, demonstrated that the usability gains were well retained in spite of the long time gap between uses. Overall, these findings highlight the potential of motion-based interaction as an intuitive interface for mobile devices.

Haptic augmented reality (AR) allows to modulate the haptic properties of a real object by providing virtual haptic feedback. We previously developed a haptic AR system wherein the stiffness of a real object can be augmented with the aid of a haptic interface. To demonstrate its potential, this paper presents a case study for medical training of breast cancer palpation. A real breast model made of soft silicone is augmented with a virtual tumor rendered inside. Haptic stimuli for the virtual tumor are generated based on a contact dynamics model identified via real measurements, without the need of geometric information on the breast. A subjective evaluation confirmed the realism and fidelity of our palpation system.

In haptic augmented reality (AR), a stimulus from a real object is augmented by a synthetic haptic signal. We previously developed a haptic AR system wherein the stiffness of a real object can be augmented with the aid of a haptic interface. This demonstration presents a case study of this technology for medical training of breast cancer palpation. A real breast model made of soft silicone is augmented with a virtual tumor rendered inside. Haptic stimuli for the virtual tumor are generated based on a contact dynamics model identified via real measurements. With our palpation system, a user can experience excellent realism comparable to that of a real breast mock-up containing a real tumor.

At present, vibration feedback is widely used to improve the limited user interface of a mobile device. Despite recent advances of miniature actuator technology, a vibration motor is still a dominant actuator for commercial mobile devices. In this paper, we present a new vibration rendering method which can enhance the identification of mobile device vibrations produced by a vibration motor. Whereas the traditional method separates vibrations in voltage applied to the motor, we partition vibrations in their perceived intensity using perceptually transparent rendering. An empirical evaluation using absolute identification showed that our rendering method can improve perception performance in terms of correct identification rate and the amount of information transfer. The results suggest that perceptually transparent rendering can contribute to increasing the number of discrete vibrations that can be used for information delivery via a mobile device, e.g., for the priorities of phone calls.

Rendering a stiff virtual surface using a force-feedback haptic interface has been one of the most classic and important research issues in haptics. In this paper, we present an initial study for a novel haptic rendering technique, named stiffness shifting, which greatly increases the perceived hardness of a virtual surface. The key idea of stiffness shifting is to use a stiffness profile that includes an instantaneous increment shortly after a contact. The algorithm is very simple, and can be easily integrated into existing haptic rendering algorithms for 3D objects. Furthermore, the perceptual performance of the algorithm is impressive; a virtual wall rendered using stiffness shifting is perceived as hard as one rendered using the common linear spring model with 2.5 times higher stiffness. This result demonstrates a great potential of stiffness shifting to be a general means for improving the perceptual quality of haptic rendering.

This paper presents a real-time framework for computationally tracking objects visually attended by the user while navigating in interactive virtual environments. In addition to the conventional bottom-up (stimulus-driven) saliency map, the proposed framework uses top-down (goal-directed) contexts inferred from the user's spatial and temporal behaviors, and identifies the most plausibly attended objects among candidates in the object saliency map. The computational framework was implemented using GPU, exhibiting high computational performance adequate for interactive virtual environments. A user experiment was also conducted to evaluate the prediction accuracy of the tracking framework by comparing objects regarded as visually attended by the framework to actual human gaze collected with an eye tracker. The results indicated that the accuracy was in the level well supported by the theory of human cognition for visually identifying single and multiple attentive targets, especially owing to the addition of top-down contextual information. Finally, we demonstrate how the visual attention tracking framework can be applied to managing the level of details in virtual environments, without any hardware for head or eye tracking.

This article presents a real-time GPU-based post-filtering method for rendering acceptable depth-of-field effects suited for virtual reality. Blurring is achieved by nonlinearly interpolating mipmap images generated from a pinhole image. Major artifacts common in the post-filtering techniques such as bilinear magnification artifact, intensity leakage, and blurring discontinuity are practically eliminated via magnification with a circular filter, anisotropic mipmapping, and smoothing of blurring degrees. The whole framework is accelerated using GPU programs for constant and scalable real-time performance required for virtual reality. We also compare our method to recent GPU-based methods in terms of image quality and rendering performance.

There are two main limitations in the conventional robot-assisted spinal fusion surgery. Since the end effector in the state of art has a role of guiding the insertion pose of a screw only, i) convenience that can be obtained when the robot intervenes in the surgery more actively could be limited, ii) The insertion pose of a screw provided by the robots could be deteriorated by surgeon's resisting force since he should insert a screw with his own hand withstanding the large reaction force transmitted through the drilling handle. To overcome those limitations, this paper proposes a novel approach for spinal fusion, wherein the robot performs the spinal fusion using the equipped end effector following surgeon's guide. We developed a dexterous small-sized the end effector that can perform previous gimleting and screwing tasks into the vertebrae. A five-DOF robot body that has kinematically-closed structure guides the insertion pose of a screw and resists strong reaction force firmly during the screwing process. Based on admittance control framework, the surgeon controls the pose of the end effector precisely to compensate induced static/dynamic errors during the operation. A torque feedback method without torque sensor that suggests the haptic information about the status of drilling is also included. The performance of the CoRASS was verified by experiments.

Abstract We present a real‐time method for rendering a depth‐of‐field effect based on the per‐pixel layered splatting where source pixels are scattered on one of the three layers of a destination pixel. In addition, the missing information behind foreground objects is filled with an additional image of the areas occluded by nearer objects. The method creates high‐quality depth‐of‐field results even in the presence of partial occlusion, without major artifacts often present in the previous real‐time methods. The method can also be applied to simulating defocused highlights. The entire framework is accelerated by GPU, enabling real‐time post‐processing for both off‐line and interactive applications.

In order to provide improved convenience for a surgeon in spinal fusion surgery, a robot system should i) closely engage in surgeon's operation using an end effector, and ii) protect the surgeon from being exposed to harmful radiation due to repeated shootings of fluoroscope. This paper proposes a bilateral teleoperation system for spinal fusion, BiTESS-II, to accomplish the goals. We developed an end effector that can substitute the surgeon's manual operation and a novel closed-loop type slave robot that can exert strong reaction force to complete gimleting and screwing tasks. Master devices are used to control the position and orientation of the slave robot and to generate haptic information identical to that of the slave side. A novel force reflection method without force sensors allowed to design the end effector simple and light. BiTESS-II is among the first human guided teleoperation system for spinal fusion with an adequate end effector. The performance of the BiTESS-II was verified by experiments.

The vibration motor is an actuator widely used for generating vibrations in haptic interaction, virtual reality, human-computer interaction, and gaming. Whereas the vibration motor has many advantages such as low price and small size, its internal structure brings a critical limitation: its output has correlated amplitude and frequency both controlled by voltage applied to the motor. Using the absolute magnitude estimation paradigm in psychophysics, we have previously obtained an Input-Output (I/O) relation from the applied voltage to vibration magnitude perceived by the user. Based on its inverse I/O relation, this paper addresses a vibration rendering method that allows the user to command the vibration motor in a perceptually transparent manner. Using the rendering method, the effects of vibration commands (in voltage) on the user's percept can be correctly understood, allowing more effective vibration design for haptic interaction. We have also implemented a prototype graphical vibration editor into which the perceptually transparent vibration rendering method is incorporated. The vibration editor can assist the user to easily design vibration effects for the vibration motor as s/he manipulates sounds with a graphical audio editor and to clearly predict the user's percept induced from the resulting vibrations as well.

This paper presents a real-time framework for computationally tracking objects visually attended by the user while navigating in interactive virtual environments. In addition to the conventional bottom-up (stimulus-driven) features, the framework also uses topdown (goal-directed) contexts to predict the human gaze. The framework first builds feature maps using preattentive features such as luminance, hue, depth, size, and motion. The feature maps are then integrated into a single saliency map using the center-surround difference operation. This pixel-level bottom-up saliency map is converted to an object-level saliency map using the item buffer. Finally, the top-down contexts are inferred from the user's spatial and temporal behaviors during interactive navigation and used to select the most plausibly attended object among candidates produced in the object saliency map. The computational framework was implemented using the GPU and exhibited extremely fast computing performance (5.68 msec for a 256X256 saliency map), substantiating its adequacy for interactive virtual environments. A user experiment was also conducted to evaluate the prediction accuracy of the visual attention tracking framework with respect to actual human gaze data. The attained accuracy level was well supported by the theory of human cognition for visually identifying a single and multiple attentive targets, especially due to the addition of top-down contextual information. The framework can be effectively used for perceptually based rendering without employing an expensive eye tracker, such as providing the depth-of-field effects and managing the level-of-detail in virtual environments.

We report position and force detection thresholds for sinusoidal waveforms in the frequency range 10-500 Hz delivered through a stylus. The participants were required to hold the stylus in a way similar to that of holding the stylus of a force-feedback device. A minishaker moved the stylus along its length so that the majority of vibrations were presented tangentially to the skin of the hand. The measured position thresholds decreased initially with an increasing stimulus frequency and formed a U-shaped curve in the high frequency region. The thresholds of high frequency vibrations were lower than those reported previously for vibrations that were perpendicular to the skin, but were similar to the thresholds reported earlier using vibrations that were tangential to the skin. A similar force threshold curve was obtained using a force sensor attached to one end of the stylus. Mechanical impedance of the skin derived from velocity estimates and force measurements indicated that the skin and tissues in the hand holding the stylus can be modeled with mass-, damper- and spring-like elements. A comparison of the mechanical impedance from the present study with those reported previously showed similar results for vibrations delivered in the tangential and normal directions to the skin

The force-constancy hypothesis states that the user of a force-feedback device maintains a constant penetration force when stroking virtual surfaces in order to perceive their topography. The hypothesis was developed to address a real-world data perceptualization problem where the perception of surface topography was distorted when the surface stiffness was nonuniform. Two experiments were conducted. In Experiment I, we recorded the penetration depths of the probe tip while the user stroked two surfaces with equal height but different stiffness values. We found that the data could be quantitatively modeled by the force-constancy hypothesis when the virtual surfaces were neither too soft nor too hard. In Experiment II, we demonstrated that given two adjacent surfaces, their perceived height difference depended on both the surface stiffness values as well as the relative heights of the surfaces. Specifically, we showed that the higher but softer surface could be perceived to be lower, at the same height, or higher than the other surface, depending on how much higher it was than the other surface. The results were consistent with the predictions of the force-constancy hypothesis. Our findings underscore the importance of understanding the interplay of haptic rendering parameters.

This study investigates the effect of update rate on the perceived quality of haptic virtual textures, focusing on one type of perceived instability called "buzzing." Buzzing refers to the high-frequency noises that may emanate from a force-feedback device during haptic texture rendering. We first present a simulation of a haptic texture rendering system that explicitly models the sampled-data nature of the system. The simulation shows that a slow haptic update rate can significantly increase the high-frequency noise in the reconstructed force command sent to the haptic interface. This noise may excite the structural resonance of the haptic interface and result in a high-frequency buzzing at the interaction tool that a user holds. A subsequent psychophysical experiment, conducted over a wide range of update rate (250 Hz - 40 kHz), has confirmed the simulation results. The results show a 278% increase in the maximum stiffness (0.325 - 0.904 N/mm over the update rates tested) that could be used for rendering perceptually "clean" virtual haptic textures. These results argue for haptic update rates that are much higher than the widely-accepted value of 1 kHz. Considering that an application requiring hard surface rendering needs a stiffness value of at least 0.8 - 1.0 N/mm, we recommend a haptic update rate in the range 5-10 kHz for perceptually stable haptic texture rendering.

This paper reports new findings of our ongoing research on perceived instability that human users frequently experience from interacting with virtual textures rendered with a force-feedback haptic interface. Our work is aimed at a better understanding of a new type of perceived instability called "aliveness", which was discovered during our previous psychophysical experiments performed using a popular texture rendering method (spring model with fixed force directions) and a common texture model (position-based sinusoidal grating). We first examine the perceptual and physical characteristics of the proximal stimuli that cause the perception of aliveness in virtual textures. It leads to the hypothesis that the virtual environment model used for computing texture-perturbing forces, not the traditional control-related instabilities of the haptic texture rendering system, is responsible for aliveness perception. We show that this conjecture is true by applying passivity-based stability theory to the position and force data measured during user interaction with virtual textures. Examples of the data where the haptic texture rendering system is passive (therefore stable) and aliveness is perceived are provided to substantiate our conclusions. Our results point to the importance of designing haptic texture rendering methods and models that are free of perceptual artifacts.

This paper reports on the results of our current research towards a qualitative characterization of the instability that a human user often perceives from a virtual textured surface rendered with a force-reflecting haptic interface. Based on our previous study of perceptual instability during haptic texture rendering, an experiment is designed and conducted in order to measure the proximal stimuli delivered by the virtual textures in terms of several physical variables (position, force, and acceleration). The measured data are analyzed in the frequency domain and characterized by sensation levels with regards to the human detection thresholds for temporal stimuli. In particular, a spectral band that conveys texture information and another one that induces instability perception are identified along with their associated sensation levels. These frequency bands excite different mechanoreceptors in the skin, and are therefore perceptually distinctive.