Unlike visual and auditory media, physical sensations are difficult to create and capture, limiting the availability of diverse haptic content. Converting common media formats like video into haptics offers a promising solution, but existing video-to-haptics methods depend on specific characteristics, such as camera motion or predefined actions, and rely on spatial haptic hardware (e.g., motion chair, haptic vest). We introduce HapticLens, an interactive method for creating haptics from video, supported by an open-source GUI and two vision algorithms. Our method works with arbitrary video content, detects subtle motion, and requires only a single vibrotactile actuator. We evaluate HapticLens through technical experiments and a study with 22 participants. Results demonstrate it supports interactive vibration design with high designer satisfaction for its usability and haptic signals’ overall quality and relevance. This work broadens the accessibility of video-driven haptics, offering a practical method to create and experience tactile content.

Environmental sounds like footsteps, keyboard typing, or dog barking carry rich information and emotional context, making them valuable for designing haptics in user applications. Existing audio-to-vibration methods, however, rely on signal-processing rules tuned for music or games and often fail to generalize across diverse sounds. To address this, we first investigated user perception of four existing audio-to-haptic algorithms, then created a data-driven model for environmental sounds. In Study 1, 34 participants rated vibrations generated by the four algorithms for 1,000 sounds, revealing no consistent algorithm preferences. Using this dataset, we trained Sound2Hap, a CNN-based autoencoder, to generate perceptually meaningful vibrations from diverse sounds with low latency. In Study 2, 15 participants rated its output higher than signal-processing baselines on both audio-vibration match and Haptic Experience Index (HXI), finding it more harmonious with diverse sounds. This work demonstrates a perceptually validated approach to audio-haptic translation, broadening the reach of sound-driven haptics.

The ability to explain complex information from chart images is vital for effective data-driven decision-making. In this work, we address the challenge of generating detailed explanations alongside answering questions about charts. We present ChartQA-X, a comprehensive dataset comprising 30,799 chart samples across four chart types, each paired with contextually relevant questions, answers, and explanations. Explanations are generated and selected based on metrics such as faithfulness, informativeness, coherence, and perplexity. Our human evaluation with 245 participants shows that model-generated explanations in ChartQA-X surpass human-written explanations in accuracy and logic and are comparable in terms of clarity and overall quality. Moreover, models fine-tuned on ChartQA-X show substantial improvements across various metrics, including absolute gains of up to 24.57 points in explanation quality, 18.96 percentage points in question-answering accuracy, and 14.75 percentage points on unseen benchmarks for the same task. By integrating explanatory narratives with answers, our approach enables agents to convey complex visual information more effectively, improving comprehension and greater trust in the generated responses.

Audio descriptions (AD) make videos accessible for blind and low vision (BLV) users by describing visual elements that cannot be understood from the main audio track. AD created by professionals or novice describers is time-consuming and offers little customization or control to BLV viewers on description length and content and when they receive it. To address this gap, we explore user-driven AI-generated descriptions, enabling BLV viewers to control both the timing and level of detail of the descriptions they receive. In a study, 20 BLV participants activated audio descriptions for seven different video genres with two levels of detail: concise and detailed. Our findings reveal differences in the preferred frequency and level of detail of ADs for different videos, participants' sense of control with this style of AD delivery, and its limitations. We discuss the implications of these findings for the development of future AD tools for BLV users.

with 18 describers (9 professionals and 9 novices) using quantitative and qualitative methods. Results show that AI support reduces repetitive work while helping professionals preserve their stylistic choices and easing the cognitive load for novices. Collaborative tags and variations show potential for providing customizations, version control, and training new describers. These findings highlight the potential of collaborative, AI-assisted tools to enhance and scale AD authorship.

Text entry for extended reality (XR) is far from perfect, and a variety of text entry techniques (TETs) have been proposed to fit various contexts of use. However, comparing between TETs remains challenging due to the lack of a consolidated collection of techniques, and limited understanding of how interaction attributes of a technique (e.g., presence of visual feedback) impact user performance. To address these gaps, this paper examines the current landscape of XR TETs by creating a database of 176 different techniques. We analyze this database to highlight trends in the design of these techniques, the metrics used to evaluate them, and how various interaction attributes impact these metrics. We discuss implications for future techniques and present TEXT: Text Entry for XR Trove, an interactive online tool to navigate our database.

Video descriptions are crucial for blind and low vision (BLV) users to access visual content. However, current artificial intelligence models for generating descriptions often fall short due to limitations in the quality of human annotations within training datasets, resulting in descriptions that do not fully meet BLV users' needs. To address this gap, we introduce VideoA11y, an approach that leverages multimodal large language models (MLLMs) and video accessibility guidelines to generate descriptions tailored for BLV individuals. Using this method, we have curated VideoA11y-40K, the largest and most comprehensive dataset of 40,000 videos described for BLV users. Rigorous experiments across 15 video categories, involving 347 sighted participants, 40 BLV participants, and seven professional describers, showed that VideoA11y descriptions outperform novice human annotations and are comparable to trained human annotations in clarity, accuracy, objectivity, descriptiveness, and user satisfaction. We evaluated models on VideoA11y-40K using both standard and custom metrics, demonstrating that MLLMs fine-tuned on this dataset produce high-quality accessible descriptions. Code and dataset are available at https://people-robots.github.io/VideoA11y/.

Designing haptic effects is a complex, time-consuming process requiring specialized skills and tools. To support haptic design, we introduce HapticGen, a generative model designed to create vibrotactile signals from text inputs. We conducted a formative workshop to identify requirements for an AI-driven haptic model. Given the limited size of existing haptic datasets, we trained HapticGen on a large, labeled dataset of 335k audio samples using an automated audio-to-haptic conversion method. Expert haptic designers then used HapticGen's integrated interface to prompt and rate signals, creating a haptic-specific preference dataset for fine-tuning. We evaluated the fine-tuned HapticGen with 32 users, qualitatively and quantitatively, in an A/B comparison against a baseline text-to-audio model with audio-to-haptic conversion. Results show significant improvements in five haptic experience (e.g., realism) and system usability factors (e.g., future use). Qualitative feedback indicates HapticGen streamlines the ideation process for designers and helps generate diverse, nuanced vibrations.

This late-breaking report presents RCareGen, a graphical interface that integrates natural language commands with RCare World, a physics simulator for robotic caregiving scenarios. RCareGen has three core modules: (1) a front-end web interface, (2) an LLM-based code generator, and (3) the RCareWorld simulation backend. The front-end web interface enables novice users to input natural language, which is translated into Python code by an LLM-based code generator. This generated code interacts with RCareWorld APIs to run the simulation backend, facilitating scene setup, modifications, simple movements, and human-robot interaction tasks. Additionally, the system supports iterative feedback, allowing users to refine scenes and tasks interactively. By simplifying simulation setup and enhancing task diversity, RCareGen introduces a novel interface that democratizes robot simulation and programming across diverse domains.

Socially assistive robots (SARs) can act as caregivers and use touch gestures to interact with people such as children with autism. However, designing a touch perception system that can reliably detect these gestures remains a challenge. To address this gap, we replicated a Do-It-Yourself (DIY) skin-like sensor and evaluated its potential to identify eight social touch gestures: Fistbump, Hitting, Holding, Poking, Squeezing, Stroking, Tapping, and Tickling. Our sensor design builds on a recent silicone-based sensor to collect spatiotemporal gesture data and a load cell to capture force information. We collected touch gestures from 20 adults in a user study. Then, we built a touch perception algorithm with a Convolutional Neural Network-Long Short-Term Memory (CNN- LSTM) model, achieving 94% in gesture classification accuracy with to-fold validation and 69 % accuracy with subject-dependent splitting with the combined touch and force data.

Multimodal learning benefits from multiple modal information, and each learned modal representations can be divided into uni-modal that can be learned from uni-modal training and paired-modal features that can be learned from cross-modal interaction. Building on this perspective, we propose a partitioner-guided modal learning framework, PgM, which consists of the modal partitioner, uni-modal learner, paired-modal learner, and uni-paired modal decoder. Modal partitioner segments the learned modal representation into uni-modal and paired-modal features. Modal learner incorporates two dedicated components for uni-modal and paired-modal learning. Uni-paired modal decoder reconstructs modal representation based on uni-modal and paired-modal features. PgM offers three key benefits: 1) thorough learning of uni-modal and paired-modal features, 2) flexible distribution adjustment for uni-modal and paired-modal representations to suit diverse downstream tasks, and 3) different learning rates across modalities and partitions. Extensive experiments demonstrate the effectiveness of PgM across four multimodal tasks and further highlight its transferability to existing models. Additionally, we visualize the distribution of uni-modal and paired-modal features across modalities and tasks, offering insights into their respective contributions.

Augmented Reality (AR) excels at altering what we see but non-visual sensations are difficult to augment. To augment non-visual sensations in AR, we draw on the visual language of comic books. Synthesizing comic studies, we create a design space describing how to use comic elements (e.g., onomatopoeia) to depict non-visual sensations (e.g., hearing). To demonstrate this design space, we built eight demos, such as speed lines to make a user think they are faster and smell lines to make a scent seem stronger. We evaluate these elements in a qualitative user study (N=20) where participants performed everyday tasks with comic elements added as augmentations. All participants stated feeling a change in perception for at least one sensation, with perceived changes detected by between four participants (touch) and 15 participants (hearing). The elements also had positive effects on emotion and user experience, even when participants did not feel changes in perception.

Mid-air ultrasound haptic technology can enhance user interaction and immersion in extended reality (XR) applications through contactless touch feedback. Yet, existing design tools for mid-air haptics primarily support creating tactile sensations (i.e., tactons) which cannot change at runtime. These tactons lack expressiveness in interactive scenarios where a continuous closed-loop response to user movement or environmental states is desirable. This paper introduces AdapTics, a toolkit featuring a graphical interface for rapid prototyping of adaptive tactons—dynamic sensations that can adjust at runtime based on user interactions, environmental changes, or other inputs. A software library and a Unity package accompany the graphical interface to enable integration of adaptive tactons in existing applications. We present the design space offered by AdapTics for creating adaptive mid-air ultrasound tactons and show the design tool can improve Creativity Support Index ratings for Exploration and Expressiveness in a user study with 12 XR and haptic designers.

Mid-air ultrasound technology offers new design opportunities for contactless tactile patterns (i.e., Tactons) in user applications. Yet, few guidelines exist for making ultrasound Tactons easy to distinguish for users. In this paper, we investigated the distinguishability of temporal parameters of ultrasound Tactons in five studies (n=72 participants). Study 1 established the discrimination thresholds for amplitude-modulated (AM) frequencies. In Studies 2–5, we investigated distinguishable ultrasound Tactons by creating four Tacton sets based on mechanical vibrations in the literature and collected similarity ratings for the ultrasound Tactons. We identified a subset of temporal parameters, such as rhythm and low envelope frequency, that could create distinguishable ultrasound Tactons. Also, a strong correlation (mean Spearman’s ρ =0.75) existed between similarity ratings for ultrasound Tactons and similarities of mechanical Tactons from the literature, suggesting vibrotactile designers can transfer their knowledge to ultrasound design. We present design guidelines and future directions for creating distinguishable mid-air ultrasound Tactons.

Advances in haptic technologies can alter how real objects feel to our touch and create the experience of haptic augmented reality (AR). However, the definition, use cases, and value to end users of such haptic AR remain unclear. Existing work is concerned with technological implementation and lacks a user-centered perspective. To address these limitations, we analyze haptic AR systems in the literature to understand what constitutes haptic AR, why we would want to alter our sense of touch, and how haptic AR interactions take place. To demonstrate the value of studying haptic AR in the context of real-world tasks and user impressions of the concept itself, we also conducted a small exploratory study with five prototypical applications of different haptic AR systems. Our analysis highlights unexplored areas for haptics and HCI researchers and the need to conduct user evaluations of the overall concept rather than just point examples.

Robots increasingly interact with humans through touch, where people are touching or being touched by robots. Yet, little is known about how such interactions shape a user’s experience. To inform future work in this area, we conduct a systematic review of 44 studies on physical human–robot interaction (pHRI). Our review examines the parameters of the touch (e.g., the role of touch, location), the experimental variations used by researchers, and the methods used to assess user experience. We identify five facets of user experience metrics from the questionnaire items and data recordings for pHRI studies. We highlight gaps and methodological issues in studying pHRI and compare user evaluation trends with the Human–Computer Interaction (HCI) literature. Based on the review, we propose a conceptual model of the pHRI experience. The model highlights the components of such touch experiences to guide the design and evaluation of physical interactions with robots and inform future user experience questionnaire development.

High-fidelity vibration actuators in recent mobile phones allow designers to crowdsource user evaluation of vibrotactile (VT) Tactons. Yet, little work has examined whether online crowdsourcing platforms can provide comparable results to lab studies. To address this question, we conducted two studies with iOS devices in the lab and crowdsourced settings. In Study I, 40 users provided pairwise similarity ratings for 12 VT Tactons that varied in their parameters (e.g., duration). In Study II, 40 new users rated pairwise similarities for 14 Tactons representing different metaphors (e.g., heartbeat). They also rated the Tactons’ match to the metaphors. In both studies, the resulting similarities and perceptual spaces strongly correlated in the lab and crowdsourced settings. Furthermore, 60% of the metaphor ratings were statistically equivalent in the two settings. We discuss the results and outline directions for future work on haptic crowdsourcing.

Ultrasound mid-air haptic technology provides a large space of design possibilities, as one can modulate the ultrasound intensity in a continuous 3D space at a high speed over time. Yet, the need for programming the patterns limits rapid ideation and testing of alternatives. We present Feellustrator, a graphical design tool for quickly creating and editing ultrasound mid-air haptics. With Feellustrator, one can create custom ultrasound patterns, layer or sequence them into complex effects, project them on the user’s hand, and export them for use in external programs (e.g., Unity). To create the tool, we interviewed 13 designers who had from a few months to several years of experience with ultrasound, then derived a set of requirements for supporting ultrasound design. We demonstrate the design power of Feellustrator through example applications and an evaluation with 15 participants. Then, we outline future directions for ultrasound haptic design.

Hugs are complex affective interactions that often include gestures like squeezes. We present six new guidelines for designing interactive hugging robots, which we validate through two studies with our custom robot. To achieve autonomy, we investigated robot responses to four human intra-hug gestures: holding, rubbing, patting, and squeezing. A Total of 32 users each exchanged and rated 16 hugs with an experimenter-controlled HuggieBot 2.0. The robot’s inflated torso’s microphone and pressure sensor collected data of the subjects’ demonstrations that were used to develop a perceptual algorithm that classifies user actions with 88% accuracy. Users enjoyed robot squeezes, regardless of their performed action, they valued variety in the robot response, and they appreciated robot-initiated intra-hug gestures. From average user ratings, we created a probabilistic behavior algorithm that chooses robot responses in real time. We implemented improvements to the robot platform to create HuggieBot 3.0 and then validated its gesture perception system and behavior algorithm with 16 users. The robot’s responses and proactive gestures were greatly enjoyed. Users found the robot more natural, enjoyable, and intelligent in the last phase of the experiment than in the first. After the study, they felt more understood by the robot and thought robots were nicer to hug.

Designing robotic hands has been an active area of research and innovation in the last decade. However, little is known about how people perceive robot hands and react to being touched by them. To inform hand design for social robots, we created a database of 73 robot hands and ran two user studies. In the first study, 160 online users rated the hands in our database. Variations in user ratings mostly centered on the perceived Comfortableness , Interestingness , and Industrialness of the hands. In a second lab-based study, users evaluated seven physical hands and had similar ratings to results from the online study. Furthermore, we did not find a significant difference in user ratings before and after the users were touched by the hands. We provide regression models that can predict user ratings from the hand features (e.g., number of fingers) and an online interface for using our robot hand database and predictive models.

Numerous techniques have been proposed for locomotion in virtual reality (VR). Several taxonomies consider a large number of attributes (e.g., hardware, accessibility) to characterize these techniques. However, finding the appropriate locomotion technique (LT) and identifying gaps for future designs in the high-dimensional space of attributes can be quite challenging. To aid analysis and innovation, we devised Locomotion Vault (https://locomotionvault.github.io/), a database and visualization of over 100 LTs from academia and industry. We propose similarity between LTs as a metric to aid navigation and visualization. We show that similarity based on attribute values correlates with expert similarity assessments (a method that does not scale). Our analysis also highlights an inherent trade-off between simulation sickness and accessibility across LTs. As such, Locomotion Vault shows to be a tool that unifies information on LTs and enables their standardization and large-scale comparison to help understand the space of possibilities in VR locomotion.

Humans rely on categories to mentally organize and understand sets of complex objects. One such set, haptic devices, has myriad technical attributes that affect user experience in complex ways. Seeking an effective navigation structure for a large online collection, we elicited expert mental categories for grounded force-feedback haptic devices: 18 experts (9 device creators, 9 interaction designers) reviewed, grouped, and described 75 devices according to their similarity in a custom card-sorting study. From the resulting quantitative and qualitative data, we identify prominent patterns of tagging versus binning, and we report 6 uber-attributes that the experts used to group the devices, favoring affordances over device specifications. Finally, we derive 7 device categories and 9 subcategories that reflect the imperfect yet semantic nature of the expert mental models. We visualize these device categories and similarities in the online haptic collection, and we offer insights for studying expert understanding of other human-centered technology.

Creating haptic experiences often entails inventing, modifying, or selecting specialized hardware. However, interaction designers are rarely engineers, and 30 years of haptic inventions are buried in a fragmented literature that describes devices mechanically rather than by potential purpose. We conceived of Haptipedia to unlock this trove of examples: Haptipedia presents a device corpus for exploration through metadata that matter to both device and interaction designers. It is a taxonomy of device attributes that go beyond physical description to capture potential utility, applied to a growing database of 105 grounded force-feedback devices, and accessed through a public visualization that links utility to morphology. Haptipedia's design was driven by both systematic review of the haptic device literature and rich input from diverse haptic designers. We describe Haptipedia's reception (including hopes it will redefine device reporting standards) and our plans for its sustainability through community participation.

When refining or personalizing a design, we count on being able to modify or move an element by changing its parameters rather than creating it anew in a different form or location—a standard utility in graphic and auditory authoring tools. Similarly, we need to tune vibrotactile sensations to fit new use cases, distinguish members of communicative icon sets, and personalize items. For tactile vibration display, however, we lack knowledge of the human perceptual mappings that must underlie such tools. Based on evidence that affective dimensions are a natural way to tune vibrations for practical purposes, we attempted to manipulate perception along three emotion dimensions (agitation,liveliness, andstrangeness) using engineering parameters of hypothesized relevance. Results from two user studies show that an automatable algorithm can increase a vibration’s perceivedagitationandlivelinessto different degrees via signal energy, while increasing its discontinuity or randomness makes it morestrange. These continuous mappings apply across diverse base vibrations; the extent of achievable emotion change varies. These results illustrate the potential for developing vibrotactile emotion controls as efficient tuning for designers and end-users.

Vibrotactile (VT) display is becoming a standard component of informative user experience, where notifications and feedback must convey information eyes-free. However, effective design is hindered by incomplete understanding of relevant perceptual qualities, together with the need for user feedback to be accessed in-situ. To access evaluation streamlining now common in visual design, we introduce proxy modalities as a way to crowdsource VT sensations by reliably communicating high-level features through a crowd-accessible channel. We investigate two proxy modalities to represent a high-fidelity tactor: a new VT visualization, and low-fidelity vibratory translations playable on commodity smartphones. We translated 10 high-fidelity vibrations into both modalities, and in two user studies found that both proxy modalities can communicate affective features, and are consistent when deployed remotely over Mechanical Turk. We analyze fit of features to modalities, and suggest future improvements.

With haptics now common in consumer devices, diversity in tactile perception and aesthetic preferences confound haptic designers. End-user customization out of example sets is an obvious solution, but haptic collections are notoriously difficult to explore. This work addresses the provision of easy and highly navigable access to large, diverse sets of vibrotactile stimuli, on the premise that multiple access pathways facilitate discovery and engagement. We propose and examine five disparate organization schemes (taxonomies), describe how we created a 120-item library with diverse functional and affective characteristics, and present VibViz, an interactive tool for end-user library navigation and our own investigation of how different taxonomies can assist navigation. An exploratory user study with and of VibViz suggests that most users gravitate towards an organization based on sensory and emotional terms, but also exposes rich variations in their navigation patterns and insights into the basis of effective haptic library navigation.

Affective response may dominate users' reactions to the synthesized tactile sensations that are proliferating in today's handheld and gaming devices, yet it is largely unmeasured, modeled or characterized. A better understanding of user perception will aid the design of tactile behavior that engages touch, with an experience that satisfies rather than intrudes. We measured 30 subjects' affective response to vibrations varying in rhythm and frequency, then examined how differences in demographic, everyday use of touch, and tactile processing abilities contribute to variations in affective response. To this end, we developed five affective and sensory rating scales and two tactile performance tasks, and also employed a published `Need for Touch' (NFT) questionnaire. Subjects' ratings, aggregated, showed significant correlations among the five scales and significant effect of the signal content (rhythm and frequency). Ratings varied considerably among subjects, but this variation did not coincide with demographic, NFT score or tactile task performance. The linkages found among the rating scales confirm this as a promising approach. The next step towards a comprehensive picture of individuals' patterns of affective response to tactile sensations entails pruning, integration and redundancy reduction of these scales, then their formal validation.