A wide range of haptic feedback is crucial for achieving high realism and immersion in virtual environments. Therefore, a multi-modal haptic interface that provides various haptic signals simultaneously is highly beneficial. This paper introduces a novel silicone fingertip actuator that is pneumatically actuated, delivering a realistic and effective haptic experience by simultaneously providing pressure, vibrotactile, and cold thermal feedback. The actuator features a design with multiple air chambers, each with controllable volume achieved through pneumatic valves connected to compressed air tanks. The lower air chamber generates pressure feedback, while the upper chamber produces vibrotactile feedback. In addition, two lateral air nozzles utilize a vortex tube to generate a cold thermal sensation. To showcase the system's capabilities, we designed two unique 3D surfaces in the virtual environment: a frozen meat surface and an abrasive icy surface. These surfaces simulate tactile perceptions of coldness, pressure, and texture. Comprehensive performance assessments and user studies were conducted to validate the actuator's effectiveness, highlighting its diverse feedback capabilities compared to traditional actuators that offer only single feedback modalities.

While model-mediated teleoperation (MMT) is an effective alternative for ensuring both transparency and stability, its potential in transmitting surface haptic texture is not yet explored. This paper introduces the first MMT framework capable of sharing surface haptic texture. The follower side collects physical signals contributing to haptic texture perception, e.g., high frequency acceleration, and streams them to the leader side. The leader side uses the signals to build and update a local measurement-based texture simulation model that reflects the remote surface. At the same time, the leader runs local simulation using the model, resulting in non-delayed, stable, and accurate feedback of texture. Considering that rendering haptic texture needs tougher real-time requirements, e.g., higher update rate and lower action-feedback latency, MMT can be a perfect platform for remote texture sharing. An initial proof-of-concept system supporting single and homogeneous surface is implemented and evaluated, demonstrating the potential of the approach.

This paper introduces a framework to predict multi-dimensional haptic attribute values that humans use to recognize the material by using the physical tactile signals (acceleration) generated when a textured surface is stroked. To this end, two spaces are established: a haptic attribute space and a physical signal space. A five-dimensional haptic attribute space is established through human adjective rating experiments with the 25 real texture samples. The physical space is constructed using tool-based interaction data from the same 25 samples. A mapping is modeled between the aforementioned spaces using a newly designed CNN-LSTM deep learning network. Finally, a prediction algorithm is implemented that takes acceleration data and returns coordinates in the haptic attribute space. A quantitative evaluation was conducted to inspect the reliability of the algorithm on unseen textures, showing that the model outperformed other similar models.

Recently, there has been an excessive congestion occurring in the driving environment because of the presence of modern gadgets inside the car and increased traffic on the roads, which has resulted in a higher demand for the visual and cognitive senses. This prompted the need to reduce the demand to make driving experience safer and more comfortable. Consequently, a novel steering wheel design for in-car controls is presented in this paper. The new design introduces dual ubiquitous touch panels embedded in the steering wheel for interaction with in-car controls and haptic feedback as positive reinforcement upon successful execution of an in-car control. There are eight different functionalities that can be controlled using the embedded touch panels. The proposed system is compared with a standard car regarding its efficacy using the NASA task load index (NASA-TLX) evaluation technique. The results showed that the proposed system significantly reduced the drivers’ visual, cognitive, and manual workload.

In this paper, we propose a measurement-based modeling framework for hyper-elastic material identification and real-time haptic rendering. We build a custom data collection setup that captures shape deformation and response forces during compressive deformation of cylindrical material samples. We collected training and testing sets of data from four silicone objects having various material profiles. We design an objective function for material parameter identification by incorporating both shape deformation and reactive forces and utilize a genetic algorithm. We adopted an optimization-based Finite Element Method (FEM) for object deformation rendering. The numerical error of simulated forces was found to be perceptually negligible.

We present an interactive virtual-reality (VR) fire extinguisher that provides both realistic viewing using a head-mounted display (HMD) and kinesthetic experiences using a pneumatic muscle and vibrotactile transducer. The VR fire extinguisher is designed to train people to use a fire extinguisher skillfully in real fire situations. We seamlessly integrate three technologies: VR, object motion tracking, and haptic feedback. A fire scene is immersed in the HMD, and a motion tracker is used to replicate a real designed object into the virtual environment to realize augmented reality. In addition, when the handle of the fire extinguisher is squeezed to release the extinguishing agent, the haptic device generates both vibrotactile and air flow tactile feedback signals, providing the same experience as that obtained while using a real fire extinguisher.

This paper presents an extended data-driven haptic rendering method capable of reproducing force responses during sliding interaction on a large surface area. The core part of the approach is a novel input variable set for data interpolation model training, which includes the position of a proxy - the contact point on a surface when undeformed. This enables us to estimate friction in both sliding and sticking states. The behavior of the proxy is simulated in real-time based on a sliding yield surface - a cone-shaped surface separating the sliding and the sticking area in the external force space, identified through an automated palpation procedure. During modeling, input data and resultant force data are collected through manual palpation of a target object, which are used to build a radial-basis interpolation model. During rendering, this interpolation model with interaction inputs estimates force responses in real-time. Performance evaluation shows that our approach achieves reasonable estimation accuracy. This work is among the first attempts that fully support sliding exploration in the data-driven paradigm.

Interaction with virtual deformable models is common in several haptic contexts, such as in medical training simulators. This paper presents a methodological procedure for the creation of such virtual models from their real-life counterparts. Both the surface geometry and the elastic parametrization of an object are reconstructed from position/force readings during an operator-assisted exploration of the object. A 3D mesh model is then generated from the surface contact points. The internal elastic modulus is found using the 3D finite element method. This modeling method is compared with two common 1D elastic models, namely Kelvin-Voigt and Hunt-Crossley. Results using three deformable homogeneous silicone samples show successful geometry reconstruction. 1D model parameterizations exhibit high variation dependent on geometry and contact location. In contrast, elastic modulus reconstruction yields a global model parameterization independent of geometry. Elastic moduli estimated in experiments correlated with their known values, and were shown to be reproducible among samples with different geometries.

Haptic augmented reality merges real and virtual feedback, allowing a user to touch the real environment augmented with virtual haptic stimuli. A key functionality of this technology is altering the haptic properties of real objects by means of virtual haptic feedback. Previously, we have developed a haptic augmentation system in which object stiffness was modulated. In this paper, we extend our framework to cover further haptic properties: friction and weight. Simple but effective algorithms for estimating and altering these properties have been developed. The first approach allows us to change the inherent friction between a tool tip and a surface to a desired one identified in an offline process. The second technique enables a user to perceive an altered weight when lifting an object at two interaction points. The performance of the proposed algorithms has been evaluated on real objects. The errors remain within reasonable bounds and compare well to the relevant perceptual Weber fractions. Limitations of our technique are identified, and possible extensions are proposed.

Haptic augmented reality (AR) allows to modulate the haptic properties of a real object by providing virtual haptic feedback. We previously developed a haptic AR system wherein the stiffness of a real object can be augmented with the aid of a haptic interface. To demonstrate its potential, this paper presents a case study for medical training of breast cancer palpation. A real breast model made of soft silicone is augmented with a virtual tumor rendered inside. Haptic stimuli for the virtual tumor are generated based on a contact dynamics model identified via real measurements, without the need of geometric information on the breast. A subjective evaluation confirmed the realism and fidelity of our palpation system.

In haptic augmented reality (AR), a stimulus from a real object is augmented by a synthetic haptic signal. We previously developed a haptic AR system wherein the stiffness of a real object can be augmented with the aid of a haptic interface. This demonstration presents a case study of this technology for medical training of breast cancer palpation. A real breast model made of soft silicone is augmented with a virtual tumor rendered inside. Haptic stimuli for the virtual tumor are generated based on a contact dynamics model identified via real measurements. With our palpation system, a user can experience excellent realism comparable to that of a real breast mock-up containing a real tumor.

Rendering a stiff virtual surface using a force-feedback haptic interface has been one of the most classic and important research issues in haptics. In this paper, we present an initial study for a novel haptic rendering technique, named stiffness shifting, which greatly increases the perceived hardness of a virtual surface. The key idea of stiffness shifting is to use a stiffness profile that includes an instantaneous increment shortly after a contact. The algorithm is very simple, and can be easily integrated into existing haptic rendering algorithms for 3D objects. Furthermore, the perceptual performance of the algorithm is impressive; a virtual wall rendered using stiffness shifting is perceived as hard as one rendered using the common linear spring model with 2.5 times higher stiffness. This result demonstrates a great potential of stiffness shifting to be a general means for improving the perceptual quality of haptic rendering.

This paper proposes to generate and provide wide field of view (FOV) augmented reality (AR) imagery by mosaicing images from smaller fields of moving views in "desktop" tangible AR (DTAR) environments. AR systems usually offer a limited FOV into the interaction space, constrained by the FOV of the camera and/or the display, which causes serious usability problems especially when the interaction space is large and many tangible props/markers are used. This problem is more apparent in DTAR environments in which an upright frontal display is used, instead of a head mounted display. This can be solved partly by placing the camera at a relatively far location or by using multiple cameras and increasing the working FOV. However, as for the former solution, the large distance between the interaction space and the fixed camera decreases the tracking and recognition reliability of the tangible markers, and the latter solution introduces significant additional set-up, cost, and computational load. Thus, we propose to use a mosaiced image to provide wide FOV AR imagery. We experimentally compare our solution, i.e. to offer the entire view of the interaction space at once, to other nominal AR set-ups. The experimental results show that, despite some amounts of visual artifacts due to the imperfect mosaicing, the proposed solution can improve task performance and usability for a typical DTAR system. Our findings should contribute to making AR systems more practical and usable for the mass.

In this paper, we have addressed the usability issue of augmented reality systems. In order to overcome the lack of view into the interaction space in nominal AR systems, we proposed to create and use a mosaiced image as a cost effective and practical solution. One of the obvious problem is that even if the view span is increased by the mosaicing, because of the fixed and limited resolution/size of the display(either HMD or upright display), the content must be viewed in smaller scale. Yang et al. has found out that increasing the geometric field of view can be tolerated to some degree without much degradation in task performance [3]. In our pilot experiment, a comparison of our mosaiced AR display to the nominal set up improved task performance when the task involved manipulation of many tangible props in a relatively wide area. Our future work is to conduct a more formal usability study and address other AR usability issues.

Hand-held devices possess a large potential as an interaction device for their today's ubiquity, and present us with an opportunity to devise new and unique ways of interaction as a smart device with multi-modal sensing and display capabilities. This paper introduces user interaction techniques (for selection, translation, scaling and rotation of objects) using a camera-equipped hand-held device such as a mobile phone or a PDA for large shared environments. We propose three intuitive interaction techniques for 2D and 3D objects in such an environment. The first approach uses the motion flow information to estimate the relative motion of the hand-held device and interact with the large display. The marker-object and marker-cursor approaches both use software markers on the interaction object or on the cursor for the various interactive tasks. The proposed interaction techniques can be further combined with many auxiliary functions and wireless services (of the hand-held devices) for seamless information sharing and exchange among multiple users. A formal usability analysis is currently on-going.