Hand–Object Interaction (HOI) is a key interaction component in Virtual Reality (VR). However, designing HOI still requires manual efforts to decide how object should be selected and manipulated, while also considering user abilities, which leads to time-consuming refinements. We present HOICraft, a VLM-based in-situ HOI authoring tool that enables part-level interaction design in VR. Here, HOICraft assists designers by recommending interactable elements from 3D objects, customizing HOI design properties, and mapping hand movement with virtual object behavior. We conducted a formative study with three expert VR designers to identify five representative HOI designs to support diverse user experiences. Building upon preference data from 20 participants, we develop an HOI mapping module with in-context learning. In a user study with 12 VR interaction designers, HOI mapping from HOICraft significantly reduced trial-and-error iterations compared to manual authoring. Finally, we assessed the usability of HOICraft, demonstrating its effectiveness for HOI design in VR.

The absence of physical information during hand-object interaction in a virtual environment diminishes realism and immersion. Kinesthetic haptic feedback has proven effective in delivering realistic object-derived haptic cues, enhancing the overall virtual reality (VR) experience. Here, we propose kinesthetic illusion through a novel application of finger tendon vibration (FTV), which creates an illusory sensation of finger movement. To effectively apply FTV for virtual object interactions, we first examine the effects of short-duration FTV (<5 s) through 3 perception studies. Based on study results, we design 6 exemplary VR scenarios, representing the overall design space of VR object interactions, and 4 different haptic rendering strategies for FTV. We evaluated these rendering methods on each VR scenario and derived a design guideline for FTV application. We then compared FTV with no vibration and simple vibration, observing that FTV enhances VR experience by providing realistic resistance on the finger, greatly improving body ownership.

We present ForceCtrl, a novel 3D hand raycasting technique that enhances pointing precision based on control-display (CD) gain controlled with user-defined pinch force. We introduce a target-agnostic approach for refining raycasting precision, overcoming limitations in human motor accuracy. User-defined pinch force, detected with surface electromyography (sEMG), enables users to easily activate or deactivate CD gain during interaction. We propose three CD gain strategies and compare them through target selection and placement tasks. Our system reduces selection errors, placement jitters, and user workload, especially for distant targets in high-difficulty tasks. These results highlight the effectiveness of applying CD gain to hand raycasting and demonstrate the potential of user-defined pinch force as a robust input modality for precise hand interaction in AR/VR.

With the advancement of haptic interfaces, recent studies have focused on enabling detailed haptic experiences in virtual reality (VR), such as fluid-haptic interaction. However, rendering forces from fluid contact often causes a high-cost computation. Given that motion-induced fluid feedback is crucial to the overall experience, we focus on hand-perceivable forces to enhance underwater haptic sensation by achieving high-fidelity rendering while considering human perceptual capabilities. We present a new multimodal (tactile and kinesthetic) haptic rendering pipeline. Here, we employ drag and added mass forces by dynamically adapting to the user's hand movement and posture with pneumatic-based haptic gloves. We defined decaying and damping effects to indicate fluid properties caused by inertia and confirmed their significant perceptual impacts compared to using only physics-based equations in a perception study. By modulating pressure variations, we reproduced fluid smoothness via exponential tactile deflation and light fluid mass via linear kinesthetic feedback. Our pipeline enabled richer and more immersive VR underwater experiences by accounting for precise hand regions and motion diversity.

Walking is among the most common human activities where the feet can gather rich tactile information from the ground. The dynamic contact between the feet and the ground generates vibration signals that can be sensed by the foot skin. While existing research focuses on foot pressure sensing and lower-limb interactions, methods of decoding tactile information from foot vibrations remain underexplored. Here, we propose a foot-equipped wearable system capable of recording wideband vibration signals during walking activities. By enabling location-based recording, our system generates maps of haptic data that encode information on ground materials, lower-limb activities, and road conditions. Its efficacy was demonstrated through studies involving 31 users walking over 18 different ground textures, achieving an overall identification accuracy exceeding 95\% (cross-user accuracy of 87\%). Our system allows pedestrians to map haptic information through their daily walking activities, which has potential applications in creating digitalized walking experiences and monitoring road conditions.

Designing haptic effects is a complex, time-consuming process requiring specialized skills and tools. To support haptic design, we introduce HapticGen, a generative model designed to create vibrotactile signals from text inputs. We conducted a formative workshop to identify requirements for an AI-driven haptic model. Given the limited size of existing haptic datasets, we trained HapticGen on a large, labeled dataset of 335k audio samples using an automated audio-to-haptic conversion method. Expert haptic designers then used HapticGen's integrated interface to prompt and rate signals, creating a haptic-specific preference dataset for fine-tuning. We evaluated the fine-tuned HapticGen with 32 users, qualitatively and quantitatively, in an A/B comparison against a baseline text-to-audio model with audio-to-haptic conversion. Results show significant improvements in five haptic experience (e.g., realism) and system usability factors (e.g., future use). Qualitative feedback indicates HapticGen streamlines the ideation process for designers and helps generate diverse, nuanced vibrations.

Recent 3D Instance Segmentation methods typically encode hundreds of instance-wise candidates with instance-specific information in various ways and refine them into final masks. However, they have yet to fully explore the benefit of these candidates. They overlook the valuable cues encoded in multiple candidates that represent different parts of the same instance, resulting in fragments. Also, they often fail to capture the precise spatial range of 3D instances, primarily due to inherent noises from sparse and unordered point clouds. In this work, to address these challenges, we propose IKNE, a novel instance-wise knowledge enhancement approach. We first introduce an Instance-wise Knowledge Aggregation (IKA) to associate scattered single instance details by optimizing correlations among candidates representing the same instance. Moreover, we present an Instance-wise Structural Guidance (ISG) to enhance the spatial understanding of candidates using structural cues from ambiguity-reduced features. Here, we utilize a simple yet effective truncated singular value decomposition algorithm to minimize inherent noises of 3D features. In our extensive experiments on large-scale datasets, ScanNetV2, ScanNet200, S3DIS, and STPLS3D, IKNE outperforms existing works. We validate the effectiveness of our modules in both kernel-based and transformer-based architectures.

The rising use of deepfakes in criminal activities presents a significant issue, inciting widespread controversy. While numerous studies have tackled this problem, most primarily focus on deepfake detection. These reactive solutions are insufficient as a fundamental approach for crimes where authenticity is disregarded. Existing proactive defenses also have limitations, as they are effective only for deepfake models based on specific Generative Adversarial Networks (GANs), making them less applicable in light of recent advancements in diffusion-based models. In this paper, we propose a proactive defense method named FaceShield, which introduces novel defense strategies targeting deepfakes generated by Diffusion Models (DMs) and facilitates defenses on various existing GAN-based deepfake models through facial feature extractor manipulations. Our approach consists of three main components: (i) manipulating the attention mechanism of DMs to exclude protected facial features during the denoising process, (ii) targeting prominent facial feature extraction models to enhance the robustness of our adversarial perturbation, and (iii) employing Gaussian blur and low-pass filtering techniques to improve imperceptibility while enhancing robustness against JPEG compression. Experimental results on the CelebA-HQ and VGGFace2-HQ datasets demonstrate that our method achieves state-of-the-art performance against the latest deepfake models based on DMs, while also exhibiting transferability to GANs and showcasing greater imperceptibility of noise along with enhanced robustness. Code is available here: https://github.com/kuai-lab/iccv25_faceshield

We propose a visuo-tactile feedback method that combines virtual hand visualization and fingertip vibrations to modulate affective roughness perception in VR. While prior work has focused on object-based textures and vibrotactile feedback, the role of visual feedback on virtual hands remains underexplored. Our approach introduces affective visual cues including line shape, motion, and color applied to hand outlines, and examines their influence on both affective responses (arousal, valence) and perceived roughness. Results show that sharp contours enhanced perceived roughness, increased arousal, and reduced valence, intensifying the emotional impact of haptic feedback. In contrast, color affected valence only, with red consistently lowering emotional positivity. These effects were especially noticeable at lower haptic intensities, where visual cues extended affective modulation into mid-level perceptual ranges. Overall, the findings highlight how integrating expressive visual cues with tactile feedback can enrich affective rendering and offer flexible emotional tuning in immersive VR interactions.

With the widespread use of virtual reality (VR), VR sickness is becoming a key barrier for users to have prolonged VR experience. To alleviate VR sickness, researchers focused on reducing sensory mismatch by aligning the visual information with other sensory cues during the VR experience. We present a wearable haptic interface enabling neck muscle vibration (NMV) with a multi-stimulus configuration. NMV's vibration on muscle spindles causes a haptic proprioceptive illusion of muscle stretch. Through NMV, we provide a simulated sensation of neck rotation to users without physically rotating the neck. For a left and right rotation on the yaw axis, we vibrated the sternocleidomastoid (SCM) muscles and splenius capitis (SC) muscles. Our lightweight interface vibrates different combinations of actuators on the left and right SCM and SC muscles to deliver multi-stimulus NMV in a desired illusory direction. We found that NMV sensation differs among individuals and is less effective during neck rotation. Based on these results, we developed a calibration and rendering process for NMV using real-time VR rotation information with varying viewpoint control. Our evaluation, which used a VR scene mimicking a common VR experience, showed that NMV effectively reduces rotation-induced VR sickness and improves the overall VR experience, such as presence.

While recent 3D instance segmentation approaches show promising results based on transformer architectures, they often fail to correctly identify instances with similar appearances. They also ambiguously determine edges, leading to multiple misclassifications of adjacent edge points. In this work, we introduce a novel framework, called EASE, to overcome these challenges and improve the perception of complex 3D instances. We first propose a semantic guidance network to leverage rich semantic knowledge from a language model as intelligent priors, enhancing the functional understanding of real-world instances beyond relying solely on geometrical information. We explicitly instruct the basic instance queries using text embeddings of each instance to learn deep semantic details. Further, we utilize the edge prediction module, encouraging the segmentation network to be edge-aware. We extract voxel-wise edge maps from point features and use them as auxiliary information for learning edge cues. In our extensive experiments on large-scale benchmarks, ScanNetV2, ScanNet200, S3DIS, and STPLS3D, our EASE outperforms existing state-of-the-art models, demonstrating its superior performance.

We propose ThermicVib, a wearable multimodal haptic glove that enhances the active perception of thermo-tactile interactions with virtual objects by integrating thermal referrals and vibrotactile phantom sensations. By fusing multimodal sensory illusions through flexible thermoelectric devices (FTED) and linear resonant actuators (LRAs), we aim to support dynamic thermal sensation adaptive to the user’s action in virtual reality (VR). Here, we developed an algorithm to render a whole-hand thermal sensation while accommodating contact and noncontact heat conditions. Based on the computed heat, we propose a simultaneous thermal and tactile rendering approach to enable dynamic thermal sensation. The user study validated the capability of our interface to support various whole-hand thermal sensations.

We present PiMForce, a novel framework that enhances hand pressure estimation by leveraging 3D hand posture information to augment forearm surface electromyography (sEMG) signals. Our approach utilizes detailed spatial information from 3D hand poses in conjunction with dynamic muscle activity from sEMG to enable accurate and robust whole-hand pressure measurements under diverse hand-object interactions. We also developed a multimodal data collection system that combines a pressure glove, an sEMG armband, and a markerless finger-tracking module. We created a comprehensive dataset from 21 participants, capturing synchronized data of hand posture, sEMG signals, and exerted hand pressure across various hand postures and hand-object interaction scenarios using our collection system. Our framework enables precise hand pressure estimation in complex and natural interaction scenarios. Our approach substantially mitigates the limitations of traditional sEMG-based or vision-based methods by integrating 3D hand posture information with sEMG signals. Video demos, data, and code are available online.

We present a novel framework for audio-guided localized image stylization. Sound often provides information about the specific context of a scene and is closely related to a certain part of the scene or object. However, existing image stylization works have focused on stylizing the entire image using an image or text input. Stylizing a particular part of the image based on audio input is natural but challenging. This work proposes a framework in which a user provides an audio input to localize the target in the input image and another to locally stylize the target object or scene. We first produce a fine localization map using an audio-visual localization network leveraging CLIP embedding space. We then utilize an implicit neural representation (INR) along with the predicted localization map to stylize the target based on sound information. The INR manipulates local pixel values to be semantically consistent with the provided audio input. Our experiments show that the proposed framework outperforms other audio-guided stylization methods. Moreover, we observe that our method constructs concise localization maps and naturally manipulates the target object or scene in accordance with the given audio input.

The recent success of the generative model shows that leveraging the multi-modal embedding space can manipu-late an image using text information. However, manipulating an image with other sources rather than text, such as sound, is not easy due to the dynamic characteristics of the sources. Especially, sound can convey vivid emotions and dynamic expressions of the real world. Here, we propose a framework that directly encodes sound into the multi-modal (image-text) embedding space and manipulates an image from the space. Our audio encoder is trained to pro-duce a latent representation from an audio input, which is forced to be aligned with image and text representations in the multi-modal embedding space. We use a direct latent op-timization method based on aligned embeddings for sound-guided image manipulation. We also show that our method can mix different modalities, i.e., text and audio, which en-rich the variety of the image modification. The experiments on zero-shot audio classification and semantic-level image classification show that our proposed model outperforms other text and sound-guided state-of-the-art methods.

We introduce a novel wearable haptic feedback device that magnifies the visual experience of virtual and augmented environments through bidirectional vibrotactile feedback driven by electromagnetic coils with permanent magnets. This device creates guidance haptic effect through magnetic attraction and repulsion. Our proof-of-concept prototype enables haptic interaction through altering position of wearable structure, vibrating with different intensity, and waveform pattern. Example applications illustrate how the proposed system promotes guided and rich haptic feedback.

In this study, we explore the effect of hand posture on haptic experience within the hand. With the wide acceptability of using the hand as an input in the XR, it is important to find out the positive and negative effects of hand postures on the haptic experience. We measured the vibration intensity of the hand using an accelerometer on various hand postures. Our results showed that distinctive hand postures alter the vibrotactile actuator’s tactile feedback across the hand.

We present HapSense, a single-volume soft haptic I/O device with uninterrupted dual functionalities of force sensing and vibrotactile actuation. To achieve both input and output functionalities, we employ a ferroelectric electroactive polymer as core functional material with a multilayer structure design. We introduce a haptic I/O hardware that supports tunable high driving voltage waveform for vibrotactile actuation while insitu sensing a change in capacitance from contact force. With mechanically soft nature of fabricated structure, HapSense can be embedded onto various object surfaces including but not limited to furniture, garments, and the human body. Through a series of experiments and evaluations, we characterized physical properties of HapSense and validated the feasibility of using soft haptic I/O with real users. We demonstrated a variety of interaction scenarios using HapSense.

The emerging simultaneous localizing and mapping (SLAM) based tracking technique allows the mobile AR device spatial awareness of the physical world. Still, smart things are not fully supported with the spatial awareness in AR. Therefore, we present Scenariot, a method that enables instant discovery and localization of the surrounding smart things while also spatially registering them with a SLAM based mobile AR system. By exploiting the spatial relationships between mobile AR systems and smart things, Scenariot fosters in-situ interactions with connected devices. We embed Ultra-Wide Band (UWB) RF units into the AR device and the controllers of the smart things, which allows for measuring the distances between them. With a one-time initial calibration, users localize multiple IoT devices and map them within the AR scenes. Through a series of experiments and evaluations, we validate the localization accuracy as well as the performance of the enabled spatial aware interactions. Further, we demonstrate various use cases through Scenariot.

This paper proposes a robust solution for accurate 3D hand pose estimation in the presence of an external object interacting with hands. Our main insight is that the shape of an object causes a configuration of the hand in the form of a hand grasp. Along this line, we simultaneously train deep neural networks using paired depth images. The object-oriented network learns functional grasps from an object perspective, whereas the hand-oriented network explores the details of hand configurations from a hand perspective. The two networks share intermediate observations produced from different perspectives to create a more informed representation. Our system then collaboratively classifies the grasp types and orientation of the hand and further constrains a pose space using these estimates. Finally, we collectively refine the unknown pose parameters to reconstruct the final hand pose. To this end, we conduct extensive evaluations to validate the efficacy of the proposed collaborative learning approach by comparing it with self-generated baselines and the state-of-the-art method.

We present iSoft, a single volume soft sensor capable of sensing real-time continuous contact and unidirectional stretching. We propose a low-cost and an easy way to fabricate such piezoresistive elastomer-based soft sensors for instant interactions. We employ an electrical impedance tomography (EIT) technique to estimate changes of resistance distribution on the sensor caused by fingertip contact. To compensate for the rebound elasticity of the elastomer and achieve real-time continuous contact sensing, we apply a dynamic baseline update for EIT. The baseline updates are triggered by fingertip contact and movement detections. Further, we support unidirectional stretching sensing using a model-based approach which works separately with continuous contact sensing. We also provide a software toolkit for users to design and deploy personalized interfaces with customized sensors. Through a series of experiments and evaluations, we validate the performance of contact and stretching sensing. Through example applications, we show the variety of examples enabled by iSoft.

We present TMotion, a self-contained 3D input that enables spatial interactions around mobile device using a magnetic sensing technique. We embed a permanent magnet and an inertial measurement unit (IMU) in a stylus. When the stylus moves around the mobile device, we obtain a continuous magnetometer readings. By numerically solving non-linear magnetic field equations with known orientation from IMU, we achieve 3D position tracking with update rate greater than 30Hz. Our experiments evaluated the position tracking accuracy, showing an average error of 4.55mm in the space of 80mm×120mm×100mm. Furthermore, the experiments confirmed the tracking robustness against orientations and dynamic tracings. In task evaluations, we verified the tracking and targeting performance in spatial interactions with users. We demonstrate example applications that highlight TMotion's interaction capability.

We present TRing, a finger-worn input device which provides instant and customizable interactions. TRing offers a novel method for making plain objects interactive using an embedded magnet and a finger-worn device. With a particle filter integrated magnetic sensing technique, we compute the fingertip's position relative to the embedded magnet. We also offer a magnet placement algorithm that guides the magnet installation location based upon the user's interface customization. By simply inserting or attaching a small magnet, we bring interactivity to both fabricated and existing objects. In our evaluations, TRing shows an average tracking error of 8.6 mm in 3D space and a 2D targeting error of 4.96 mm, which are sufficient for implementing average-sized conventional controls such as buttons and sliders. A user study validates the input performance with TRing on a targeting task (92% accuracy within 45 mm distance) and a cursor control task (91% accuracy for a 10 mm target). Furthermore, we show examples that highlight the interaction capability of our approach.

Building from our previous work we explore HandiMate, a robotics kit which enables users to construct and animate their toys using everyday craft materials [32]. The kit contains eight joint modules, a tablet interface and a glove controller. Unlike popular kits, HandiMate does not rely on manufactured parts to construct the toy. Rather this open ended platform engages users to pursue interest driven activities using everyday objects, such as cardboard, construction paper, and spoons. These crafted parts are then fastened together using Velcro to the joint modules and animated using the glove as the controller. In this paper, we discuss the results from two user studies which were designed to understand the affinity of HandiMate among children. The first study reveals that children rated the HandiMate kit as gender-neutral, appealing equally to both female and male students. The second study discusses the benefits of engaging children in engineering design with HandiMate, which has been observed to bring out children's tacit physics-based engineering knowledge and facilitate learning.

We present SOFTii, a flexible input system for topography design and continuous control via external force. Our intent is to provide a tactile metaphor for pressure-based surface input. In this study, two prototypes of SOFTii have been fabricated: (a) The first prototype has one pressure surface for topography design with everyday tangible objects, (b) the second prototype, having two force input surfaces, performs as a deformable controller for video games and continuous shape modeling using a SVM algorithm. Both prototypes of SOFTii are constructed by layering Polymethylsiloxane (PDMS), ITO coated PET film, and conductive fabric and foam. The layer configuration allows the capturing of local pressure on the SOFTii surface via distributed electrodes. Here we further discuss the implementation of the device with possible usage scenarios.

We introduce TIMMi, a textile input device for mobile interactions. TIMMi is worn on the index finger to provide a multimodal sensing input metaphor. The prototype is fabricated on a single layer of textile where the conductive silicone rubber is painted and the conductive threads are stitched. The sensing area comprises of three equally spaced dots and a separate wide line. Strain and pressure values are extracted from the line and three dots, respectively via voltage dividers. Regression analysis is performed to model the relationship between sensing values and finger pressure and bending. A multi-level thresholding is applied to capture different levels of finger bending and pressure. A temporal position tracking algorithm is implemented to capture the swipe gesture. In this preliminary study, we demonstrate TIMMi as a finger-worn input device with two applications: controlling music player and interacting with smartglasses.

We present BendID, a bendable input device that recognizes the location, magnitude and direction of its deformation. We use BendID to provide users with a tactile metaphor for pressure based input. The device is constructed by layering an array of indium tin oxide (ITO)-coated PET film electrodes on a Polymethylsiloxane (PDMS) sheet, which is sandwiched between conductive foams. The pressure values that are interpreted from the ITO electrodes are classified using a Support Vector Machine (SVM) algorithm via the Weka library to identify the direction and location of bending. A polynomial regression model is also employed to estimate the overall magnitude of the pressure from the device. A model then maps these variables to a GUI to perform tasks. In this preliminary paper, we demonstrate this device by implementing it as an interface for 3D shape bending and a game controller.

This paper presents the development of 2-DOF robotic exoskeleton for upper limb rehabilitation after stroke. We focused on the powered exoskeleton which consists of a cable-driven mechanism actuated by brushless motors. Strain gages measured torques which loaded elbow flexion/extension and wrist pronation/supination axes. To begin each trials, sEMG(surface Electromyography) was used to detect the voluntary movements of subjects.