The perceived realism of virtual body illusions, such as arm elongation, is dependent on congruent sensory feedback. This paper explores the effects of visio-haptic stimulation on the sense of embodiment in Virtual Reality (VR). We conducted three experiments. The first identified the optimal parameters for vibrotactile feedback by varying stimulation location and motor activation duration (250, 500, 750 ms) during a fixed visual task of 40% arm elongation over 18 seconds. We found that sequential stimulations along the entire arm with a 500ms duration per motor were the most effective. From this, we derived a formula to generalize the optimal haptic timing based on elongation velocity. The formula was then validated in two follow-up experiments: one varying the elongation percentage (26% to 80%) and another varying the elongation duration (9s to 27s). Our findings offer validated guidelines for designing more immersive haptic experiences for virtual body deformations.

The ability to manipulate and physically feel virtual objects without any real object being present and without equipping the user has been a long-standing goal in virtual reality (VR). Emerging ultrasound mid-air haptics (UMH) technology could potentially address this challenge, as it enables remote tactile stimulation of unequipped users. However, to date, UMH has received limited attention in the field of haptic exploration and manipulation in virtual environments. Existing work has primarily focused on interactions requiring a single hand and thus the delivery of unimanual haptic feedback. Despite being fundamental to a large part of haptic interactions with our environments, bimanual tasks have rarely been studied in the field of UMH interaction in VR. In this paper, we propose the use of non-coplanar mid-air haptic devices for providing simultaneous tactile feedback to both hands during bimanual VR manipulation. We discuss coupling schemes and haptic rendering algorithms for providing bimanual haptic feedback in bimanual interactions with virtual environments. We then present two human participant studies, assessing the benefits of bimanual ultrasound haptic feedback in a two-handed grasping and holding task and in a shape exploration task. Results suggest that the use of multiple non-coplanar UMH devices could be an interesting approach for enriching unencumbered haptic manipulation in virtual environments.

Ultrasound mid-air haptic (UMH) interfaces focus acoustic energy to generate high-pressure focal points in mid-air, creating tactile sensations. When these focal points are modulated over time, they can render tactile patterns with various properties in a 3D workspace. While UMH has been used to render curves and 2D shapes, such as tactile icons, rendering 3D objects in full is nowadays an open problem, as it would require more powerful devices. To overcome this limitation, designers typically only render a representation of the intersection between the user’s hand and the virtual object. However, there is no consensus in the literature on how best to compute this intersection.

Wearable haptic devices can modify the haptic perception of an object touched directly by the finger in a portable and unobtrusive way. In this paper, we investigate whether such wearable haptic augmentations are perceived differently in Augmented Reality (AR) vs. Virtual Reality (VR) and when touching with a virtual hand instead of one’s own hand. We first designed a system for real-time rendering of vibrotactile virtual textures without constraints on hand movements, integrated with an immersive visual AR/VR headset. We then conducted a psychophysical study with 20 participants to evaluate the haptic perception of virtual roughness textures on a real surface touched directly with the finger (1) without visual augmentation, (2) with a realistic virtual hand rendered in AR, and (3) with the same virtual hand in VR. On average, participants overestimated the roughness of haptic textures when touching with their real hand alone and underestimated it when touching with a virtual hand in AR, with VR in between. Exploration behaviour was also slower in VR than with real hand alone, although subjective evaluation of the texture was not affected. We discuss how the perceived visual delay of the virtual hand may produce this effect.

Abstract High‐fidelity digital human modeling has become crucial in various applications, including gaming, visual effects and virtual reality. Despite the significant impact of eyelashes on facial aesthetics, their reconstruction and modeling have been largely unexplored. In this paper, we introduce the first data‐driven generative model of eyelashes based on semantic features. This model is derived from real data by introducing a new 3D eyelash reconstruction method based on multi‐view images. The reconstructed data is made available which constitutes the first dataset of 3D eyelashes ever published. Through an innovative extraction process, we determine the features of any set of eyelashes, and present detailed descriptive statistics of human eyelashes shapes. The proposed eyelashes model, which exclusively relies on semantic parameters, effectively captures the appearance of a set of eyelashes. Results show that the proposed model enables interactive, intuitive and realistic eyelashes modeling for non‐experts, enriching avatar creation and synthetic data generation pipelines.

Rich, informative and realistic haptic feedback is key to enhancing Virtual Reality (VR) manipulation. Tangible objects provide convincing grasping and manipulation interactions with haptic feedback of e.g., shape, mass and texture properties. But these properties are static, and cannot respond to interactions in the virtual environment. On the other hand, vibrotactile feedback provides the opportunity for delivering dynamic cues rendering many different contact properties, such as impacts, object vibrations or textures. Handheld objects or controllers in VR are usually restricted to vibrating in a monolithic fashion. In this article, we investigate how spatialiazing vibrotactile cues within handheld tangibles could enable a wider range of sensations and interactions. We conduct a set of perception studies, investigating the extent to which spatialization of vibrotactile feedback within tangible objects is possible as well as the benefits of proposed rendering schemes leveraging multiple actuators in VR. Results show that vibrotactile cues from localized actuators can be discriminated and are beneficial for certain rendering schemes.

In this paper, we propose a physics-based robot controller to deform a soft object toward a desired 3D shape using a limited number of handling points. For this purpose, the shape of the deformable object is represented using Fourier descriptors. We derive the analytical relation that provides the variation of the Fourier coefficients as a function of the movements of the handling points by considering a mass-spring model (MSM). A control law is then designed from this relation. Since the MSM provides an approximation of the object behavior, which in practice can lead to a drift between the object and its model, an online realignment of the model with the real object is performed by tracking its surface from data provided by a remote RGB-D camera. Simulation results validate the approach for the case where many points interact on a 2D soft object while experimental results obtained with two robotic arms demonstrate the autonomous shaping of a 3D soft object.

Ultrasound mid-air haptic (UMH) devices are a novel tool for haptic feedback, capable of providing localized vibrotactile stimuli to users at a distance. UMH applications largely rely on generating tactile shape outlines on the users’ skin. Here we investigate how to achieve sensations of continuity or gaps within such two-dimensional curves by studying the perception of pairs of amplitude-modulated focused ultrasound stimuli. On the one hand, we aim to investigate perceptual effects that may arise from providing simultaneous UMH stimuli. On the other hand, we wish to provide perception-based rendering guidelines for generating continuous or discontinuous sensations of tactile shapes. Finally, we hope to contribute toward a measure of the perceptually achievable resolution of UMH interfaces. We performed a user study to identify how far apart two focal points need to be to elicit a perceptual experience of two distinct stimuli separated by a gap. Mean gap detection thresholds were found at 32.3-mm spacing between focal points, but a high within- and between-subject variability was observed. Pairs spaced below 15 mm were consistently (>95%) perceived as a single stimulus, while pairs spaced 45 mm apart were consistently (84%) perceived as two separate stimuli. To investigate the observed variability, we resort to acoustic simulations of the resulting pressure fields. These show a non-linear evolution of actual peak pressure spacing as a function of nominal focal point spacing. Beyond an initial threshold in spacing (between 15 and 18 mm), which we believe to be related to the perceived size of a focal point, the probability of detecting a gap between focal points appears to linearly increase with spacing. Our work highlights physical interactions and perceptual effects to consider when designing or investigating the perception of UMH shapes.

Ultrasound mid-air haptic (UMH) devices can remotely render vibrotactile shapes on the skin of unequipped users, e.g., to draw haptic icons or render virtual object shapes. Spatio-temporal modulation (STM), the state-of-the-art UMH shape-rendering method, provides large freedom in shape design and produces the strongest possible stimuli for this technology. Yet, STM shapes are often reported to be blurry, complicating shape identification. Dynamic tactile pointers (DTP) were recently introduced as a technique to overcome this issue. By tracing a contour with an amplitude-modulated focal point, they significantly improve shape identification accuracy over STM, but at the cost of much lower stimulus intensity. Building upon this, we propose spatio-temporally-modulated Tactile Pointers (STP), a novel method for rendering clearer and sharper UMH shapes while at the same time producing strong vibrotactile sensations. We ran two human participant experiments, which show that STP shapes are perceived as significantly stronger than DTP shapes, while shape identification accuracy is significantly improved over STM and on par with that obtained with DTP. Our work has implications for effective shape rendering with UMH and provides insights that could inform future psychophysical investigation into vibrotactile shape perception in UMH.

In this paper, we propose a physics-based model of suction phenomenon to achieve simulation of deformable objects like suction cups. Our model uses a constraint-based formulation to simulate the variations of pressure inside suction cups. The respective internal pressures are represented as pressure constraints which are coupled with anti-interpenetration and friction constraints. Furthermore, our method is able to detect multiple air cavities using information from collision detection. We solve the pressure constraints based on the ideal gas law while considering several cavity states. We test our model with a number of scenarios reflecting a variety of uses, for instance, a spring loaded jumping toy, a manipulator performing a pick and place task, and an octopus tentacle grasping a soda can. We also evaluate the ability of our model to reproduce the physics of suction cups of varying shapes, lifting objects of different masses, and sliding on a slippery surface. The results show promise for various applications such as the simulation in soft robotics and computer animation.

In this special issue ofIEEE Transactions on Visualization and Computer Graphics (TVCG), we are pleased to present the journal papers from the 22nd IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2023), which will be held as a hybrid conference between October 16 and 20, 2023 in Sydney, Australia. ISMAR continues the over twenty-year long tradition of IWAR, ISMR, and ISAR, and is the premier conference for Mixed and Augmented Reality in the world.

In this paper, we present a complete pipeline for positioning a feature point of a soft object to a desired 3D position, by acting on a different manipulation point using a robotic manipulator. For that purpose, the analytic relation between the feature point displacement and the robot motion is derived using a coarse mass-spring model (MSM), while taking into consideration the propagation delay introduced by a MSM. From this modeling step, a novel closed-loop controller is designed for performing the positioning task. To get rid of the model approximations, the object is tracked in real-time using a RGB-D sensor, thus allowing to correct on-line any drift between the object and its model. Our model-based and vision-based controller is validated in real experiments for two different soft objects and the results show promising performance in terms of accuracy, efficiency and robustness.

The use of electrotactile feedback in Virtual Reality (VR) has shown promising results for providing tactile information and sensations. While progress has been made to provide custom electrotactile feedback for specific interaction tasks, it remains unclear which modulations and rendering algorithms are preferred in rich interaction scenarios. In this paper, we propose a unified tactile rendering architecture and explore the most promising modulations to render finger interactions in VR. Based on a literature review, we designed six electrotactile stimulation patterns/effects (EFXs) striving to render different tactile sensations. In a user study (N=18), we assessed the six EFXs in three diverse finger interactions: 1) tapping on a virtual object; 2) pressing down a virtual button; 3) sliding along a virtual surface. Results showed that the preference for certain EFXs depends on the task at hand. No significant preference was detected for tapping (short and quick contact); EFXs that render dynamic intensities or dynamic spatio-temporal patterns were preferred for pressing (continuous dynamic force); EFXs that render moving sensations were preferred for sliding (surface exploration). The results showed the importance of the coherence between the modulation an the interaction being performed and the study proved the versatility of electrotactile feedback and its efficiency in rendering different haptic information and sensations.

Abstract Even if the appearance and geometry of the human eye have been extensively studied during the last decade, the geometrical correlation between gaze direction, eyelids aperture and eyelids shape has not been empirically modeled. In this paper, we propose a data‐driven approach for capturing and modeling the subtle features of the human eye region, such as the inner eye corner and the skin bulging effect due to globe orientation. Our approach consists of an original experimental setup to capture the eye region geometry variations combined with a 3D reconstruction method. Regarding the eye region capture, we scanned 55 participants doing 36 eyes poses. To animate a participant's eye region, we register the different poses to a vertex wise correspondence before blending them in a trilinear fashion. We show that our 3D animation results are visually pleasant and realistic while bringing novel eye features compared to state of the art models.

While user's perception and performance are predominantly examined independently in virtual reality, the Action-Specific Perception (ASP) theory postulates that the performance of an individual on a task modulates this individual's spatial and time perception pertinent to the task's components and procedures. This paper examines the association between performance and perception and the potential effects that tactile feedback modalities could generate. This paper reports a user study (N=24), in which participants performed a standardized Fitts's law target acquisition task by using three feedback modalities: visual, visuo-electrotactile, and visuo-vibrotactile. The users completed 3 Target Sizes × 2 Distances × 3 feedback modalities = 18 trials. The size perception, distance perception, and (movement) time perception were assessed at the end of each trial. Performance-wise, the results showed that electrotactile feedback facilitates a significantly better accuracy compared to vibrotactile and visual feedback, while vibrotactile provided the worst accuracy. Electrotactile and visual feedback enabled a comparable reaction time, while the vibrotactile offered a substantially slower reaction time than visual feedback. Although amongst feedback types the pattern of differences in perceptual aspects were comparable to performance differences, none of them was statistically significant. However, performance indeed modulated perception. Significant action-specific effects on spatial and time perception were detected. Changes in accuracy modulate both size perception and time perception, while changes in movement speed modulate distance perception. Also, the index of difficulty was found to modulate all three perceptual aspects. However, individual differences appear to affect the magnitude of action-specific effects. These outcomes highlighted the importance of haptic feedback on performance, and importantly the significance of action-specific effects on spatial and time perception in VR, which should be considered in future VR studies.

Ultrasound mid-air haptic interfaces can display highly reconfigurable vibrotactile shapes in mid-air for human-computer interaction (HCI) applications. The choice of stimulus shape, spatial, temporal and modulation parameters yields a complex design space, yet relatively little is known about the impact of these design choices on perceived stimulus properties. We define the combination of a spatial discretization of an abstract shape and a set of rules for the temporal display order and intensity modulation of the resulting points as a sampling strategy. We developed DOLPHIN, an open-source framework to aid in designing mid-air stimuli. DOLPHIN allows the study of the impact of rendering parameters on perceived stimulus properties. This platform-agnostic framework standardizes stimulus descriptions as a step toward more replicability and easier communication in the field. It enables reproduction of stimuli between perceptual experiments and ensures stimuli used in applications correspond to those evaluated in prior perceptual studies. We validated DOLPHIN’s usability by conducting a user study assessing the impact of sampling strategy design on curvature discrimination for dynamic mid-air haptic stimuli. The Weber fractions for just-noticeable differences (JNDs) in curvature were found to range between 1 and 1.4, yet no significant effect of the number of spatial sampling points on curvature discrimination was found. This result shows a practical use-case for DOLPHIN and provides insight into rendering mid-air haptic curvature.

Virtual steering techniques enable users to navigate in larger Virtual Environments (VEs) than the physical workspace available. Even though these techniques do not require physical movement of the users (e.g. using a joystick and the head orientation to steer towards a virtual direction), recent work observed that users might unintentionally move in the physical workspace while navigating, resulting in Unintended Positional Drift (UPD). This phenomenon can be a safety issue since users may unintentionally reach the physical boundaries of the workspace while using a steering technique. In this context, as a necessary first step to improve the design of navigation techniques minimizing the UPD, this paper aims at analyzing and modeling the UPD during a virtual navigation task. In particular, we characterize and analyze the UPD for a dataset containing the positions and orientations of eighteen users performing a virtual slalom task using virtual steering techniques. Participants wore a head-mounted display and had to follow three different sinusoidal-like trajectories (with low, medium and high curvature) using a torso-steering navigation technique. We analyzed the performed motions and proposed two UPD models: the first based on a linear regression analysis and the second based on a Gaussian Mixture Model (GMM) analysis. Then, we assessed both models through a simulation-based evaluation where we reproduced the same navigation task using virtual agents. Our results indicate the feasibility of using simulation-based evaluations to study UPD. The paper concludes with a discussion of potential applications of the results in order to gain a better understanding of UPD during steering and therefore improve the design of navigation techniques by compensating for UPD.

In this article, we propose an approach towards an infinite surface haptic display. Our approach, named ENcountered-Type ROtating Prop Approach (ENTROPiA) is based on a cylindrical spinning prop attached to a robot's end-effector serving as an encountered-type haptic display (ETHD). This type of haptic display permits the users to have an unconstrained, free-hand contact with a surface being provided by a robotic device for the users' to encounter a surface to be touched. In our approach, the sensation of touching a virtual surface is given by an interaction technique that couples with the sliding movement of the prop under the users' finger by tracking their hand location and establishing a path to be explored. This approach enables large motion for a larger surface rendering, permits to render multi-textured haptic feedback, and leverages the ETHD approach introducing large motion and sliding/friction sensations. As a part of our contribution, a proof of concept was designed for illustrating our approach. A user study was conducted to assess the perception of our approach showing a significant performance for rendering the sensation of touching a large flat surface. Our approach could be used to render large haptic surfaces in applications such as rapid prototyping for automobile design.

We combine tracking information from a tangible object instrumented with capacitive sensors and an optical tracking system, to improve contact rendering when interacting with tangibles in VR. A human-subject study shows that combining capacitive sensing with optical tracking significantly improves the visuohaptic synchronization and immersion of the VR experience.

In this paper, we propose the ADVISEd (Active Deformation through VIsual SErvoing) method, a novel model-free deformation servoing method able to deform a soft object towards a desired shape. ADVISEd relies on an online estimation of the deformation Jacobian that relates the motion of the robot end-effector to the deformation behavior of the object. The estimation is based on a weighted least-squares minimization with a sliding window. The robustness of the method to observation noise is ensured using an eigenvalue-based confidence criterion. The ADVISEd method is validated through comparisons with a model-based and a model-free state-of-the-art methods. Two experimental setups are proposed to compare the methods, one to perform a marker-based active shaping task and one to perform several marker-less active shaping and shape preservation tasks. Experiments showed that our approach can interactively control the deformations of an object in different tasks while ensuring better robustness to external perturbations than the state-of-the-art methods.

In this paper, we propose a novel method to simultaneously track the deformation of soft objects and estimate their elasticity parameters. The tracking of the deformable object is performed by combining the visual information captured by a RGB-D sensor with interactive Finite Element Method simulations of the object. The visual information is more particularly used to distort the simulated object. In parallel, the elasticity parameter estimation minimizes the error between the tracked object and a simulated object deformed by the forces that are measured using a force sensor. Once the elasticity parameters are estimated, our tracking algorithm can be used to estimate the deformation forces applied to an object without the use of a force sensor. We validated our method on several soft objects with different shape complexities. Our evaluations show the ability of our method to estimate the elasticity parameters as well as its use to estimate the forces applied to a deformable object without any force sensor. These results open novel perspectives to better track and control deformable objects during robotic manipulations.

Learning is usually performed by observing real robot executions. Physics-based simulators are a good alternative for providing highly valuable information while avoiding costly and potentially destructive robot executions. We present a novel approach for learning the probabilities of symbolic robot action outcomes. This is done leveraging different environments, such as physics-based simulators, in execution time. To this end, we propose MENID (Multiple Environment Noise Indeterministic Deictic) rules, a novel representation able to cope with the inherent uncertainties present in robotic tasks. MENID rules explicitly represent each possible outcomes of an action, keep memory of the source of the experience, and maintain the probability of success of each outcome. We also introduce an algorithm to distribute actions among environments, based on previous experiences and expected gain. Before using physics-based simulations, we propose a methodology for evaluating different simulation settings and determining the least time-consuming model that could be used while still producing coherent results. We demonstrate the validity of the approach in a dismantling use case, using a simulation with reduced quality as simulated system, and a simulation with full resolution where we add noise to the trajectories and some physical parameters as a representation of the real system.

Encountered-Type Haptic Displays (ETHDs) represent a field of haptic displays with the premise of not using any type of actuator directly in contact with the user skin, thus providing an alternative integration of haptic displays in virtual environments. In this paper, we present novel interaction techniques (ITs) dedicated to ETHDs. The techniques aim at addressing the issues commonly presented for these devices such as limited contact areas, lags and unexpected collisions with the user. First, our paper proposes a design framework based on several parameters defining the interactive process between user and ETHD (input, movement control, displacement and contact). Five techniques based on different ramifications of the design space framework were conceived, respectively named: Swipe, Drag, Clutch, Bubble and Follow. Then, a use-case scenario was designed to depict the usage of these techniques on the task of touching and coloring a wide, flat surface. Finally, a user study based on the coloring task was conducted to assess the performance and user experience for each IT. Results were in favor of Drag and Clutch techniques which are based on manual surface displacement, absolute position selection and intermittent contact interaction. Taken together our results and design methodology pave the way to the design of future ITs for ETHDs in virtual environments.

While suction cups are widely used in Robotics, the literature is underdeveloped when it comes to the modelling and simulation of the suction phenomenon. In this paper, we present a novel physically-based approach to simulate the behavior of active suction cups. Our model relies on a novel formulation which assumes the pressure exerted on a suction cup during active control is based on constraint resolution. Our algorithmic implementation uses a classification process to handle the contacts during the suction phenomenon of the suction cup on a surface. Then, we formulate a convenient way for coupling the pressure constraint with the multiple contact constraints. We propose an evaluation of our approach through a comparison with real data, showing the ability of our model to reproduce the behavior of suction cups. Our approach paves the way for improving the design as well as the control of robotic actuators based on suction cups such as vaccum grippers.

In minimally-invasive procedures like biopsy, the physician has to insert a needle into the tissues of a patient to reach a target. Currently, this task is mostly performed manually and under visual guidance. However, manual needle insertion can result in a large final positioning error of the tip that might lead to misdiagnosis and inadequate treatment. A way to solve this limitation is to use shared control; a gesture-assistance paradigm that combines the cognitive skills of the operator with the precision, stamina and repeatability of a robotic or haptic device. In this paper, we propose to assist the physician with a haptic device that holds the needle and generates mechanical guides during the phase of manual needle pre-positioning. In the latter, the physician has to place the tip of the needle on a planned entry point, with a pre-defined angle of incidence. From this pre-operative information and also from intra-operative measurements, we propose to generate haptic cues, known as virtual fixtures, to guide the physician towards the desired position and orientation of the needle. It takes the form of five haptic guides, each one implementing virtual fixtures. We conducted a user study where those guides were compared to the unassisted reference gesture. The most constraining guide, in terms of assisted degrees of freedom, was highlighted as the one that provides the best results in terms of performance and user experience.

In this paper, we present the design of ten different 3D user interactions using passive haptics and embedded in an escape game scenario in which users have to escape from a pyramid in a limited time. Our solution is innovative by its modularity, allowing interactions with virtual objects using tangible props manipulated either directly using the hands and feet or indirectly through a single prop held in the hand, in order to perform several interactions with the virtual environment (VE). We also propose a navigation technique based on the “impossible spaces” design, allowing users to naturally walk through several overlapping rooms of the VE. All together, our different interaction techniques allow the users to solve several enigmas built into a challenging scenario inside a pyramid.

Tangible objects are a simple yet effective way for providing haptic sensations in Virtual Reality. For achieving a compelling illusion, there should be a good correspondence between what users see in the virtual environment and what they touch in the real world. The haptic features of the tangible object should indeed match those of the corresponding virtual one in terms of, e.g., size, local shape, mass, texture. A straightforward solution is to create perfect tangible replicas of all the virtual objects in the scene. However, this is often neither feasible nor desirable. This paper presents an innovative approach enabling the use of few tangible objects to render many virtual ones. The proposed algorithm analyzes the available tangible and virtual objects to find the best grasps in terms of matching haptic sensations. It starts by identifying several suitable pinching poses on the considered tangible and virtual objects. Then, for each pose, it evaluates a series of haptically-salient characteristics. Next, it identifies the two most similar pinching poses according to these metrics, one on the tangible and one on the virtual object. Finally, it highlights the chosen pinching pose, which provides the best matching sensation between what users see and touch. The effectiveness of our approach is evaluated through a user study. Results show that the algorithm is able to well combine several haptically-salient object features to find convincing pinches between the given tangible and virtual objects.

This paper studies the combination of tangible objects and wearable haptics for improving the display of stiffness sensations in virtual environments. Tangible objects enable to feel the general shape of objects, but they are often passive or unable to simulate several varying mechanical properties. Wearable haptic devices are portable and unobtrusive interfaces able to generate varying tactile sensations, but they often fail at providing convincing stiff contacts and distributed shape sensations. We propose to combine these two approaches in virtual and augmented reality (VR/AR), becoming able of arbitrarily augmenting the perceived stiffness of real/tangible objects by providing timely tactile stimuli at the fingers. We developed a proof-of-concept enabling to simulate varying elasticity/stiffness sensations when interacting with tangible objects by using wearable tactile modules at the fingertips. We carried out a user study showing that wearable haptic stimulation can well alter the perceived stiffness of real objects, even when the tactile stimuli are not delivered at the contact point. We illustrated our approach both in VR and AR, within several use cases and different tangible settings, such as when touching surfaces, pressing buttons and pistons, or holding an object. Taken together, our results pave the way for novel haptic sensations in VR/AR by better exploiting the multiple ways of providing simple, unobtrusive, and low-cost haptic displays.

Does it feel the same when you touch an object in Augmented Reality (AR) or in Virtual Reality (VR)? In this paper we study and compare the haptic perception of stiffness of a virtual object in two situations: (1) a purely virtual environment versus (2) a real and augmented environment. We have designed an experimental setup based on a Microsoft HoloLens and a haptic force-feedback device, enabling to press a virtual piston, and compare its stiffness successively in either Augmented Reality (the virtual piston is surrounded by several real objects all located inside a cardboard box) or in Virtual Reality (the same virtual piston is displayed in a fully virtual scene composed of the same other objects). We have conducted a psychophysical experiment with 12 participants. Our results show a surprising bias in perception between the two conditions. The virtual piston is on average perceived stiffer in the VR condition compared to the AR condition. For instance, when the piston had the same stiffness in AR and VR, participants would select the VR piston as the stiffer one in 60% of cases. This suggests a psychological effect as if objects in AR would feel "softer" than in pure VR. Taken together, our results open new perspectives on perception in AR versus VR, and pave the way to future studies aiming at characterizing potential perceptual biases.

The hands of virtual characters are highly complex 3D models that can be tedious and time-consuming to animate with current methods. This paper introduces THING, a novel tablet-based approach that leverages multi-touch interaction for a quick and precise control of a 3D hand's pose. The flexion/extension and abduction/adduction of the virtual fingers can be controlled for each finger individually or for several fingers in parallel through sliding motions on the tablet's surface. We designed two variants of THING: (1) MobileTHING, which maps the spatial location and orientation of the tablet to that of the virtual hand, and (2) DesktopTHING, which combines multi-touch controls of fingers with traditional mouse controls for the hand's global position and orientation. We compared the usability of THING against mouse-only controls and a data glove in two controlled experiments. Results show that DesktopTHING was significantly preferred by users while providing performance similar to data gloves. Together, these results could pave the way to the introduction of novel hybrid user interfaces based on tablets and mice in future animation pipelines.

In this paper, we present a novel approach for tracking a deformable anatomical target within 3D ultrasound volumes. Our method is able to estimate deformations caused by the physiological motions of the patient. The displacements of moving structures are estimated from an intensity-based approach combined with a physically-based model and has therefore the advantage to be less sensitive to the image noise. Furthermore, our method does not use any fiducial marker and has real-time capabilities. The accuracy of our method is evaluated on real data acquired from an organic phantom. The validation is performed on different types of motions comprising rigid and non-rigid motions. Thus, our approach opens novel possibilities for computer-assisted interventions where deformable organs are involved.

Haptic feedback is known to improve 3D interaction in virtual environments but current haptic interfaces remain complex and tailored to desktop interaction. In this paper, we introduce the “Elastic-Arm”, a novel approach for incorporating haptic feedback in immersive virtual environments in a simple and cost-effective way. The Elastic-Arm is based on a body-mounted elastic armature that links the user's hand to her shoulder. As a result, a progressive resistance force is perceived when extending the arm. This haptic feedback can be incorporated with various 3D interaction techniques and we illustrate the possibilities offered by our system through several use cases based on well-known examples such as the Bubble technique, Redirected Touching and pseudo-haptics. These illustrative use cases provide users with haptic feedback during selection and navigation tasks but they also enhance their perception of the virtual environment. Taken together, these examples suggest that the Elastic-Arm can be transposed in numerous applications and with various 3D interaction metaphors in which a mobile hap-tic feedback can be beneficial. It could also pave the way for the design of new interaction techniques based on “human-scale” egocentric haptic feedback.

Interactive dexterous manipulation of virtual objects remains a complex challenge that requires both appropriate hand models and accurate physically-based simulation of interactions. In this paper, we propose an approach based on novel aggregate constraints for simulating dexterous grasping using soft fingers. Our approach aims at improving the computation of contact mechanics when many contact points are involved, by aggregating the multiple contact constraints into a minimal set of constraints. We also introduce a method for non-uniform pressure distribution over the contact surface, to adapt the response when touching sharp edges. We use the Coulomb-Contensou friction model to efficiently simulate tangential and torsional friction. We show through different use cases that our aggregate constraint formulation is well-suited for simulating interactively dexterous manipulation of virtual objects through soft fingers, and efficiently reduces the computation time of constraint solving.

In this paper we introduce a novel approach for stereoscopic rendering of virtual environments with a wide Field-of-View (FoV) up to 360°. Handling such a wide FoV implies the use of non-planar projections and generates specific problems such as for rasterization and clipping of primitives. We propose a novel pre-clip stage specifically adapted to geometric approaches for which problems occur with polygons spanning across the projection discontinuities. Our approach integrates seamlessly with immersive virtual reality systems as it is compatible with stereoscopy, head-tracking, and multi-surface projections. The benchmarking of our approach with different hardware setups could show that it is well compliant with real-time constraint, and capable of displaying a wide range of FoVs. Thus, our geometric approach could be used in various VR applications in which the user needs to extend the FoV and apprehend more visual information.

Imagine you are facing a mirror, seeing at the same time both your real body and a virtual display of your brain in activity and perfectly superimposed to your real image “inside your real skull”. In this paper, we introduce a novel augmented reality paradigm called “Mind-Mirror” which enables the experience of seeing “through your own head”, visualizing your brain “in action and in situ”. Our approach relies on the use of a semi-transparent mirror positioned in front of a computer screen. A virtual brain is displayed on screen and automatically follows the head movements using an optical face-tracking system. The brain activity is extracted and processed in real-time with the help of an electroencephalography cap (EEG) worn by the user. A rear view is also proposed thanks to an additional webcam recording the rear of the user's head. The use of EEG classification techniques enables to test a Neurofeedback scenario in which the user can train and progressively learn how to control different mental states, such as “concentrated” versus “relaxed”. The results of a user study comparing a standard visualization used in Neurofeedback to our approach showed that the Mind-Mirror could be successfully used and that the participants have particularly appreciated its innovation and originality. We believe that, in addition to applications in Neurofeedback and Brain-Computer Interfaces, the Mind-Mirror could also be used as a novel visualization tool for education, training or entertainment applications.

In complex scenes with many objects, collision detection plays a key role in the simulation performance. This is particularly true in fracture simulation for two main reasons. One is that fracture fragments tend to exhibit very intensive contact, and the other is that collision detection data structures for new fragments need to be computed on the fly. In this paper, we present novel collision detection algorithms and data structures for real-time simulation of fracturing rigid bodies. We build on a combination of well-known efficient data structures, namely, distance fields and sphere trees, making our algorithm easy to integrate on existing simulation engines. We propose novel methods to construct these data structures, such that they can be efficiently updated upon fracture events and integrated in a simple yet effective self-adapting contact selection algorithm. Altogether, we drastically reduce the cost of both collision detection and collision response. We have evaluated our global solution for collision detection on challenging scenarios, achieving high frame rates suited for hard real-time applications such as video games or haptics. Our solution opens promising perspectives for complex fracture simulations involving many dynamically created rigid objects.

In this paper we study how the visual animation of a self-avatar can be artificially modified in real-time in order to generate different haptic perceptions. In our experimental setup, participants could watch their self-avatar in a virtual environment in mirror mode while performing a weight lifting task. Users could map their gestures on the self-animated avatar in real-time using a Kinect. We introduce three kinds of modification of the visual animation of the self-avatar according to the effort delivered by the virtual avatar: 1) changes on the spatial mapping between the user’s gestures and the avatar, 2) different motion profiles of the animation, and 3) changes in the posture of the avatar (upper-body inclination). The experimental task consisted of a weight lifting task in which participants had to order four virtual dumbbells according to their virtual weight. The user had to lift each virtual dumbbells by means of a tangible stick, the animation of the avatar was modulated according to the virtual weight of the dumbbell. The results showed that the altering the spatial mapping delivered the best performance. Nevertheless, participants globally appreciated all the different visual effects. Our results pave the way to the exploitation of such novel techniques in various VR applications such as sport training, exercise games, or industrial training scenarios in single or collaborative mode.

Haptic interaction with different types of materials in the same scene is a challenging task, mainly due to the specific coupling mechanisms that are usually required for either fluid, deformable or rigid media. Dynamically-changing materials, such as melting or freezing objects, present additional challenges by adding another layer of complexity in the interaction between the scene and the haptic proxy. In this paper, we address these issues through a common simulation framework, based on Smoothed-Particle Hydrodynamics, and enable haptic interaction simultaneously with fluid, elastic and rigid bodies, as well as their melting or freezing. We introduce a mechanism to deal with state changes, allowing the perception of haptic feedback during the process, and a set of dynamic mechanisms to enrich the interaction through the proxy. We decouple the haptic and visual loops through a dual GPU implementation. An initial evaluation of the approach is performed through performance and feedback measurements, as well as a small user study assessing the capability of users to recognize the different states of matter they interact with.

In this paper, we present a method to localize and track manually inserted needles in real-time using a three-dimensional ultrasound probe mounted on a robotized arm. The system tracks the needle using online image processing. We first propose a new algorithm capable of robustly detecting a needle from the moment it is inserted, without any a priori information on the insertion direction. By combining the random sample consensus (RANSAC) algorithm with Kalman filtering in closed loop, we achieve robust real-time tracking of the needle. In addition, we propose a control scheme to automatically guide the ultrasonic probe in order to keep the needle within the field of view, while aligning its axis with the ultrasound beam. This method will ease the insertion of the needle by the operator, and allow the development of autonomous needle insertion by medical robots.

Brain-Computer Interfaces (BCI) introduce a novel way of interacting with real and virtual environments by directly exploiting cerebral activity. However in most setups using a BCI, the user is explicitly asked to remain as motionless as possible, since muscular activity is commonly admitted to add noise and artifacts in brain electrical signals. Thus, as for today, people have been rarely let using other classical input devices such as mice or joysticks simultaneously to a BCI-based interaction. In this paper, we present an experimental study on the influence of manipulating an input device such as a standard computer mouse on the performance of a BCI system. We have designed a 2-class BCI which relies on Alpha brainwaves to discriminate between focused versus relaxed mental activities. The study uses a simple virtual environment inspired by the well-known Pac-Man videogame and based on BCI and mouse controls. The control of mental activity enables to eat pellets in a simple 2D virtual maze. Different levels of motor activity achieved with the mouse are progressively introduced in the gameplay: 1) no motor activity (control condition), 2) a semi-automatic motor activity, and 3) a highly-demanding motor activity. As expected the BCI performance was found to slightly decrease in presence of motor activity. However, we found that the BCI could still be successfully used in all conditions, and that relaxed versus focused mental activities could still be significantly discriminated even in presence of a highly-demanding mouse manipulation. These promising results pave the way to future experimental studies with more complex mental and motor activities, but also to novel 3D interaction paradigms that could mix BCI and other input devices for virtual reality and videogame applications.

We introduce the Elastic Images, a novel pseudo-haptic feedback technique which enables the perception of the local elasticity of images without the need of any haptic device. The proposed approach focus on whether visual feedback is able to induce a sensation of stiffness when the user interacts with an image using a standard mouse. The user, when clicking on a Elastic Image, is able to deform it locally according to its elastic properties. To reinforce the effect, we also propose the generation of procedural shadows and creases to simulate the compressibility of the image and several mouse cursors replacements to enhance pressure and stiffness perception. A psychophysical experiment was conducted to quantify this novel pseudo-haptic perception and determine its perceptual threshold (or its Just Noticeable Difference). The results showed that users were able to recognize up to eight different stiffness values with our proposed method and confirmed that it provides a perceivable and exploitable sensation of elasticity. The potential applications of the proposed approach range from pressure sensing in product catalogs and games, or its usage in graphical user interfaces for increasing the expressiveness of widgets.

Virtual walking, a fundamental task in Virtual Reality (VR), is greatly influenced by the locomotion interface being used, by the specificities of input and output devices, and by the way the virtual environment is represented. No matter how virtual walking is controlled, the generation of realistic virtual trajectories is absolutely required for some applications, especially those dedicated to the study of walking behaviors in VR, navigation through virtual places for architecture, rehabilitation and training. Previous studies focused on evaluating the realism of locomotion trajectories have mostly considered the result of the locomotion task (efficiency, accuracy) and its subjective perception (presence, cybersickness). Few focused on the locomotion trajectory itself, but in situation of geometrically constrained task. In this paper, we study the realism of unconstrained trajectories produced during virtual walking by addressing the following question: did the user reach his destination by virtually walking along a trajectory he would have followed in similar real conditions? To this end, we propose a comprehensive evaluation framework consisting on a set of trajectographical criteria and a locomotion model to generate reference trajectories. We consider a simple locomotion task where users walk between two oriented points in space. The travel path is analyzed both geometrically and temporally in comparison to simulated reference trajectories. In addition, we demonstrate the framework over a user study which considered an initial set of common and frequent virtual walking conditions, namely different input devices, output display devices, control laws, and visualization modalities. The study provides insight into the relative contributions of each condition to the overall realism of the resulting virtual trajectories.

We present a novel physically based approach for simulating realistic brittle fracture of impacting bodies in real time. Our method is mainly composed of two novel parts: 1) a fracture initiation method based on modal analysis, and 2) a fast energy-based fracture propagation algorithm. We propose a way to compute the contact durations and the contact forces between stiff bodies to simulate the damped deformation wave that is responsible for fracture initiation. As a consequence, our method naturally takes into account the damping properties of the bodies as well as the contact properties to simulate the fracture. To obtain a complete fracture pipeline, we present an efficient way to generate the fragments and their geometric surfaces. These surfaces are sampled on the edges of the physical mesh, to visually represent the actual fracture surface computed. As shown in our results, the computation time performances and realism of our method are well suited for physically based interactive applications.

In this paper we introduce novel 'Camera Motions' (CMs) to improve the sensations related to locomotion in virtual environments (VE). Traditional Camera Motions are artificial oscillating motions applied to the subjective viewpoint when walking in the VE, and they are meant to evoke and reproduce the visual flow generated during a human walk. Our novel camera motions are: (1) multistate, (2) personified, and (3) they can take into account the topography of the virtual terrain. Being multistate, our CMs can account for different states of locomotion in VE namely: walking, but also running and sprinting. Being personified, our CMs can be adapted to avatar's physiology such as to its size, weight or training status. They can then take into account avatar's fatigue and recuperation for updating visual CMs accordingly. Last, our approach is adapted to the topography of the VE. Running over a strong positive slope would rapidly decrease the advance speed of the avatar, increase its energy loss, and eventually change the locomotion mode, influencing the visual feedback of the camera motions. Our new approach relies on a locomotion simulator partially inspired by human physiology and implemented for a real-time use in Desktop VR. We have conducted a series of experiments to evaluate the perception of our new CMs by naive participants. Results notably show that participants could discriminate and perceive transitions between the different locomotion modes, by relying exclusively on our CMs. They could also perceive some properties of the avatar being used and, overall, very well appreciated the new CMs techniques. Taken together, our results suggest that our new CMs could be introduced in Desktop VR applications involving first-person navigation, in order to enhance sensations of walking, running, and sprinting, with potentially different avatars and over uneven terrains, such as for: training, virtual visits or video games.

Have you ever dreamed of having eyes in the back of your head? In this paper we present a novel display device called FlyVIZ which enables humans to experience a real-time 360° vision of their surroundings for the first time. To do so, we combine a panoramic image acquisition system (positioned on top of the user's head) with a Head-Mounted Display (HMD). The omnidirectional images are transformed to fit the characteristics of HMD screens. As a result, the user can see his/her surroundings, in real-time, with 360° images mapped into the HMD field-ofview. We foresee potential applications in different fields where augmented human capacity (an extended field-of-view) could benefit, such as surveillance, security, or entertainment. FlyVIZ could also be used in novel perception and neuroscience studies.

Abstract A common weathering effect is the appearance of cracks due to material fractures. Previous exemplar‐based aging and weathering methods have either reused images or sought to replicate observed patterns exactly. We introduce a new approach to exemplar‐based modeling that creates weathered patterns on synthetic objects by matching thestatisticsof fracture patterns in a photograph. We present a user study to determine which statistics are correlated to visual similarity and how they are perceived by the user. We then describe a revised physically‐based fracture model capable of producing a wide range of crack patterns at interactive rates. We demonstrate how a Bayesian optimization method can determine the parameters of this model so it can produce a pattern with the same key statistics as an exemplar. Finally, we present results using our approach and various exemplars to produce a variety of fracture effects in synthetic renderings of complex environments. The speed of the fracture simulation allows interactive previews of the fractured results and its application on large scale environments.

Immersive spaces such as 4-sided displays with stereo viewing and high-quality tracking provide a very engaging and realistic virtual experience. However, walking is inherently limited by the restricted physical space, both due to the screens (limited translation) and the missing back screen (limited rotation). In this paper, we propose three novel locomotion techniques that have three concurrent goals: keep the user safe from reaching the translational and rotational boundaries; increase the amount of real walking and finally, provide a more enjoyable and ecological interaction paradigm compared to traditional controller-based approaches. We notably introduce the "Virtual Companion", which uses a small bird to guide the user through VEs larger than the physical space. We evaluate the three new techniques through a user study with travel-to-target and path following tasks. The study provides insight into the relative strengths of each new technique for the three aforementioned goals. Specifically, if speed and accuracy are paramount, traditional controller interfaces augmented with our novel warning techniques may be more appropriate; if physical walking is more important, two of our paradigms (extended Magic Barrier Tape and Constrained Wand) should be preferred; last, fun and ecological criteria would favor the Virtual Companion.

Haptic interaction with virtual objects is a major concern in the virtual reality field. There are many physically-based efficient models that enable the simulation of a specific type of media, e.g. fluid volumes, deformable and rigid bodies. However, combining these often heterogeneous algorithms in the same virtual world in order to simulate and interact with different types of media can be a complex task. In this paper, we propose the first haptic rendering technique for the simulation and the interaction with multistate media, namely fluids, deformable bodies and rigid bodies, in real-time and with 6DoF haptic feedback. Based on the Smoothed-Particle Hydrodynamics (SPH) physical model for all three types of media, our method avoids the complexity of dealing with different algorithms and their coupling. We achieve high update rates while simulating a physically-based virtual world governed by fluid and elasticity theories, and show how to render interaction forces and torques through a 6DoF haptic device.

We often interact with fluids in our daily life, either through tools such as when holding a glass of water or directly with our body when we swim or we wash our hands. Multimodal interactions with virtual fluids would greatly improve the simulations realism, particularly through haptic interaction. However, achieving realistic, stable, and real-time force feedback from fluids is particularly challenging. In this work, we propose a novel approach that allows real-time six Degrees of Freedom (DoF) haptic interaction with fluids of variable viscosity. Our haptic rendering technique, based on a Smoothed-Particle Hydrodynamics physical model, provides a realistic haptic feedback through physically based forces. 6DoF haptic interaction with fluids is made possible thanks to a new coupling scheme and a unified particle model, allowing the use of arbitrary-shaped rigid bodies. Particularly, fluid containers can be created to hold fluid and hence transmit to the user force feedback coming from fluid stirring, pouring, shaking, and scooping, to name a few. Moreover, we adapted an existing visual rendering algorithm to meet the frame rate requirements of the haptic algorithms. We evaluate and illustrate the main features of our approach through different scenarios, highlighting the 6DoF haptic feedback and the use of containers.

The concept of affordance, introduced by the psychologist James Gibson, can be defined as the functional utility of an object, a surface or an event. The purpose of this article was to evaluate the perception of affordances in virtual environments (VE). In order to test this perception, we considered the affordances for standing on a virtual slanted surface. The participants were asked to judge whether a virtual slanted surface supported upright stance. The perception was investigated by manipulating the texture of the slanted surface (Wooden texture vs. Ice texture). Results showed an effect of the texture: the perceptual boundary (or critical angle) with the Ice texture was significantly lower than with the Wooden texture. These results reveal that perception of affordances for standing on a slanted surface in virtual reality is possible and comparable to previous studies conducted in real environments.

The Walking-In-Place interaction technique was introduced to navigate infinitely in 3D virtual worlds by walking in place in the real world. The technique has been initially developed for users standing in immersive setups and was built upon sophisticated visual displays and tracking equipments.

In this paper, we present an experiment whose goal is to investigate the role of sound and vision in the recognition of different surface profiles in a walking scenario. Fifteen subjects participated to two within-subjects experiments where they were asked to interact with a desktop system simulating bumps, holes and flat surfaces by means of audio, visual and audio-visual cues. Results of the first experiment show that participants are able to successfully identify the surface profiles provided through the proposed audio-visual techniques. Results of a second experiment in which conflictual audiovisual stimuli were presented, reveal that for some of the proposed visual effects the visual feedback is dominant on the auditory one, while for the others the role of dominance is inverted.

In most virtual reality simulations the virtual world is larger than the real walking workspace. The workspace is often bounded by the tracking area or the display devices. This paper describes a novel interaction metaphor called the Magic Barrier Tape, which allows a user to navigate in a potentially infinite virtual scene while confined to a restricted walking workspace. The technique relies on the barrier tape metaphor and its do not cross implicit message by surrounding the walking workspace with a virtual barrier tape in the scene. Therefore, the technique informs the user about the boundaries of his walking workspace, providing an environment safe from collisions and tracking problems. It uses a hybrid position/rate control mechanism to enable real walking inside the workspace and rate control navigation to move beyond the boundaries by pushing on the virtual barrier tape. It provides an easy, intuitive and safe way of navigating in a virtual scene, without break of immersion. Two experiments were conducted in order to evaluate the Magic Barrier Tape by comparing it to two state-of-the-art navigation techniques. Results showed that the Magic Barrier Tape was faster and more appreciated than the compared techniques, while being more natural and less tiring. Considering it can be used in many different virtual reality systems, it is an interaction metaphor suitable for many different applications, from the entertainment field to training simulations scenarios.

Until recently, computer-aided medical interventions (CAMI) and medical robotics have focused on rigid and nondeformable anatomical structures. Nowadays, special attention is paid to soft tissues, raising complex issues due to their mobility and deformation. Mini-invasive digestive surgery was probably one of the first fields where soft tissues were handled through the development of simulators, tracking of anatomical structures and specific assistance robots. However, other clinical domains, for instance urology, are concerned. Indeed, laparoscopic surgery, new tumour destruction techniques (e.g., HIFU, radiofrequency, or cryoablation), increasingly early detection of cancer, and use of interventional and diagnostic imaging modalities, recently opened new challenges to the urologist and scientists involved in CAMI. This resulted in the last five years in a very significant increase of research and developments of computer-aided urology systems. In this paper, we propose a description of the main problems related to computer-aided diagnostic and therapy of soft tissues and give a survey of the different types of assistance offered to the urologist: robotization, image fusion, surgical navigation. Both research projects and operational industrial systems are discussed