Naturalistic human touch expression can be an emotionally potent modality, and promising as an informative but unobtrusive input into machine-learning models of affect. However, application-ready emotion-aware technologies must be trained on labeled samples of authentic (felt) emotion of a significant range of intensity and valence. These may come at significant, even traumatic, personal cost for any modality. We examined (a) performance of the novel touch modality, and (b) how a care-centered protocol might manage personal burden. Participants (N=5; 3 team members), shared autobiographical stories that elicited powerful emotional dynamics, in 1-3 sessions each (total 10). During storytelling, incidental touch was captured on a pillow-mounted custom flexible 10×10-taxel pressure sensor, alongside physiological signals; then labeled with multiple passes of rich, multimodal self-reports. Protocol, study and analysis design prioritized reflexivity and participant experience.Accuracy:Participant-specific, touch-only models predicted emotion direction (trajectory slope) with 65.2$\pm$16.3% accuracy (2s windows; chance 25%, physiology-only models 64.1$\pm$16.9%), confirming the value of this unobtrusivechannel.Personal cost:Qualitative analysis contributed an extensive picture of the emotional toll of generating such data, but also some benefits. We offer recommendations for sustainable ethical sourcing of affective data which balance personalization, performance, therapeutic insight and participant care.

Computational emotion recognition relies on observable expressions. However, negative situations can evoke regulation mechanisms that obscure and mask emotional experiences, often by smiling. As smiles are typically associated with positive emotions, this mismatch of emotional experience and expression may lead to misinterpretations by most current algorithmic affective computing approaches. To improve computational modeling of real-life experiences and expressions in negative social situations, we explore connections between smile appearance and function, incorporating participants' rich personal self-reports into ground truth labels for their expressions. We present an empirically grounded smile corpus of 199 smiles that is based on a) recordings of N = 30 participants in negative social situations that are analyzed regarding smile morphology and b) a category system of smile functions based on participants‘ self-reports. In a computational model, we used cleaned corpus data of 183 unique smile instances to classify five smile function categories based on observable nonverbal signals, with results benchmarked at above chance. Applying a theory- and data-driven approach, our analyses confirm a complex relationship between internal smile functions and observable signals. Finally, we discuss smile functions in negative social situations, including ‘despising’, ‘provoking’, and 'kiss my ass'-smiles.

Humans physically express emotion by modulating parameters that register on mammalian skin mechanoreceptors, but are unavailable in current touch-sensing technology. Greater sensory richness combined with data on affect-expression composition is a prerequisite to estimating affect from touch, with applications including physical human-robot interaction. To examine shear alongside more easily captured normal stresses, we tailored recent capacitive technology to attain performance suitable for affective touch, creating a flexible, reconfigurable and soft 36-taxel array that detects multitouch normal and 2-dimensional shear at ranges of 1.5kPa-43kPa and ± 0.3-3.8kPa respectively, wirelessly at 43Hz (1548 taxels/s). In a deep-learning classification of 9 gestures (N=16), inclusion of shear data improved accuracy to 88%, compared to 80% with normal stress data alone, confirming shear stress’s expressive centrality. Using this rich data, we analyse the interplay of sensed-touch features, gesture attributes and individual differences, propose affective-touch sensing requirements, and share technical considerations for performance and practicality.

Children Living with Health Complexity (CLHC), whose multiple, severe and chronic conditions differentiate them from those with “complicated” conditions, rely life-long on a vast and ever-shifting array of care providers. Their parent caregivers face a fragmented health care system, disconnected medical records, inter-stakeholder communication barriers and an impenetrable accumulation of documentation – from mundane to life-critical, and largely on paper. They must coordinate care while organizing, tracking and transmitting trends in many health parameters to myriad care providers. We engaged with parent caregivers of CLHC from 12 families through an iterative, 3-phase co-design process to understand their needs for a future digital management system. We share our deepened understanding of their information-centered challenges, a set of principles for how design best-practices need to shift when targeting this acutely high-needs group, and a medium-fidelity prototype user interface which from the ground-up prioritizes caregiver-centered data integration and humanization of the child and family, as well as integrated health record access.

The global pandemic exposed serious drawbacks in relying on communication modalities in which social touch, however important, is absent. Considerable research has explored haptic technologies for sensing or displaying social touch and influencing affective state, for wellness, social communication, emotion regulation, and affect therapy. However, this Affective Haptic System design (AHSD) work varies widely in purpose and origin discipline, making it difficult to perceive overall progress and identify primary obstacles to practical deployment. We conducted a scoping review and conceptual analysis with a design lens, identifying 110 papers from the last decade in 11 ACM and IEEE venues that regularly attract AHSD work. Our analysis identified 38 dimensions within 8 facets: demographic, theoretical grounding, impact, system specification, usage specification, ethical consideration, technology, and evaluation. We visualize trends, disciplinary mixing, and topical focus over time, and highlight major advances while pinning down crucial gaps that can be addressed in the future.

Practical affect recognition needs to be efficient and unobtrusive in interactive contexts. One approach to a robust realtime system is to sense and automatically integrate multiple nonverbal sources. We investigated how users’touch, and secondarilygaze, perform as affect-encoding modalities during physical interaction with a robot pet, in comparison to more-studied biometric channels. To elicit authentically experienced emotions, participants recounted two intense memories of opposing polarity inStressed-RelaxedorDepressed-Excitedconditions. We collected data (N=30) from a touch sensor embedded under robot fur (force magnitude and location), a robot-adjacent gaze tracker (location), and biometric sensors (skin conductance, blood volume pulse, respiration rate). Cross-validation of Random Forest classifiers achieved best-case accuracy for combined touch-with-gaze approaching that of biometric results: where training and test sets include adjacent temporal windows, subject-dependent prediction was 94% accurate. In contrast, subject-independent Leave-One-participant-Out predictions resulted in 30% accuracy (chance 25%). Performance was best where participant information was available in both training and test sets. Addressing computational robustness for dynamic, adaptive realtime interactions, we analyzed subsets of our multimodal feature set, varying sample rates and window sizes. We summarize design directions based on these parameters for this touch-based, affective, and hard, realtime robot interaction application.

Touch is valued for supporting emotional bonds. How can people access its warmth and nuance remotely, when tech-mediated proxies are so different from direct touch? We assessed the viability of haptic animations as affect-embedded tactile messages, highlighting findings which demonstrate how crucial relationship and shared history is in influencing these expressions in design and interpretation. To investigate haptic messaging, we first identified a set of 10 common emotion-imbued scenarios by surveying 201 people in distance relationships. Then, using a novel prototype of a wearable spatial vibrotactile display, 10 intimate dyads designed 167 haptic encodings matching the provided scenarios plus 17 user-defined “wildcards”. A week later, 21 individuals interpreted sentiment from encodings designed by themselves, a partner or a stranger. We examined design strategies, engagement, and compared humanversusmachine interpretation accuracy. A striking finding was participants’ facile use of shared context when it was available, building on “inside stories” to communicate subtle meanings with high effectiveness despite the unfamiliar medium, and doing so with evident fun. We analyze recognition accuracy and share insights on what it might take to make interpersonal haptic messaging work.

Many emotion classification and prediction approaches focus on emotion state, defined as static and single-valued. In contrast, our in-body experience is of sensations that can quickly evolve, consistent with scientific evidence of physiological regulation mechanisms. Can we reframe classification to estimate dynamic emotion parameters at interactive rates? For insight into dynamic emotion characteristics, we developed a multipass labelling protocol to capture controlled yet genuine emotion evolution elicited as 16 participants played a tense video game. We analyze and align multiple self-report outputs, inspect the signals for emotion dynamics, and consider label metaphors of position and angle — “where I am” vs. “where I'm going”. Finally, we reflect on the benefits and drawbacks of such a protocol for developing models of fast-evolving emotion.

The availability of voice-user interfaces (VUIs) has grown dramatically in recent years. As more capable systems invite higher expectations, the conversational interactions that VUIs support introduce ambiguity in accountability: a user's or system's obligation or willingness to be responsible for the outcome of user-delegated tasks. When misconstrued, impact ranges from inconvenience to deadly harm. This project explores how users' accountability perceptions and expectations can be managed in voice interaction with smart home appliances. To explore links between degree of automation, system accountability and user satisfaction, we identified key design factors for VUI design through an exploratory study, articulated them in video prototypes of four new VUI mechanisms showing a user commanding an advanced appliance and encountering a problem, and deployed them in a second study. We found that participants were more satisfied with automated systems, but also saw them as more accountable. Results suggest that this perceived system accountability was reduced in mechanisms that justified their recommendations. Our findings motivate the development of transitional mechanisms which gradually provide less guidance as users learn to operate the system.

Rapid iteration is crucial to effective prototyping; yet making certain objects - large, smoothly curved and/or of specific material - requires specialized equipment or considerable time. To improve access to casting such objects, we developed FoldMold: a low-cost, simply-resourced and eco-friendly technique for creating scalable, curved mold shapes (any developable surface) with wax-stiffened paper. Starting with a 3D digital shape, we define seams, add bending, joinery and mold-strengthening features, and "unfold" the shape into a 2D pattern, which is then cut, assembled, wax-dipped and cast with materials like silicone, plaster, or ice. To access the concept's full power, we facilitated digital pattern creation with a custom Blender add-on. We assessed FoldMold's viability, first with several molding challenges in which it produced smooth, curved shapes far faster than 3D printing would; then with a small user study that confirmed automation usability. Finally, we describe a range of opportunities for further development.

No abstract available.

We describe the design and deployment of Calmer, a technology that simulates key aspects of maternal skin-to-skin holding for prematurely born infants: its inspiration, approach, physical design, and introduction into the Neonatal Intensive Care Unit. Maternal skin-to-skin holding can mitigate neonatal pain during medical procedures by as much as 50%, which can improve weight gain, sleep and later development. However, parents cannot always be present, and some infants are too fragile to be held. Interventions targeting this gap could be perceived as supplanting the mother in this intimate role, exposing her to depression and endangering her maternal bond. Over 10 years, we iteratively developed Calmer and demonstrated infant health benefit in a randomized clinical trial. Here, we report and reflect on pursuing this goal in a socially and technologically complex context: constraints, strategies, features, reception of the device, and surprises, such as leading to mothers feeling channeled rather than replaced.

When sketching, we must choose between paper (expressive ease, ruler and eraser) and computational assistance (parametric support, a digital record). PHysically Assisted SKetching provides both, with a pen that displays force constraints with which the sketcher interacts as they draw on paper. Phasking provides passive, "bound" constraints (like a ruler); or actively "brings" the sketcher along a commanded path (e.g., a curve), which they can violate for creative variation. The sketcher modulates constraint strength (control sharing) by bearing down on the pen-tip. Phasking requires untethered, graded force-feedback, achieved by modifying a ballpoint drive that generates force through rolling surface contact. To understand phasking's viability, we implemented its interaction concepts, related them to sketching tasks and measured device performance. We assessed the experience of 10 sketchers, who could understand, use and delight in phasking, and who valued its control-sharing and digital twinning for productivity, creative control and learning to draw.

Humans rely on categories to mentally organize and understand sets of complex objects. One such set, haptic devices, has myriad technical attributes that affect user experience in complex ways. Seeking an effective navigation structure for a large online collection, we elicited expert mental categories for grounded force-feedback haptic devices: 18 experts (9 device creators, 9 interaction designers) reviewed, grouped, and described 75 devices according to their similarity in a custom card-sorting study. From the resulting quantitative and qualitative data, we identify prominent patterns of tagging versus binning, and we report 6 uber-attributes that the experts used to group the devices, favoring affordances over device specifications. Finally, we derive 7 device categories and 9 subcategories that reflect the imperfect yet semantic nature of the expert mental models. We visualize these device categories and similarities in the online haptic collection, and we offer insights for studying expert understanding of other human-centered technology.

To create emotionally expressive robots, designers of human-robot interaction routinely translate emotion theories into instruments through which we estimate, quantify and analyze human emotional responses to robot behaviour. Pragmatically, we often use straightforward models such as Russell's circumplex, treating emotion as a single point in a two-dimensional space. However, this simple metaphor and its consequent representations omit many aspects of real emotional experience, can lead to erroneous data and may undermine computational models that rely on them. Problems with emotion representations currently prevalent in human-robot interaction fall into three categories: (1)Representations are static and singular, whereas real emotions can be dynamic, multi-valued, uncertain or conflicting. (2)The framing of an interaction is unspecified (i.e., in an affective rating task: which part of an interaction involving multiple parties and perspectives the participant is meant to consider). (3) Participant responses captured with instruments and methods that are not well-understood by experimenters nor participants produce data that is hard to interpret. We propose alternative emotion representations to account for dynamic emotions inherent in interactive contexts; scrutinize framing ambiguities in study tasks and argue for mixed-methods approaches to achieve shared understanding of emotion representations between participants and researchers.

Creating haptic experiences often entails inventing, modifying, or selecting specialized hardware. However, interaction designers are rarely engineers, and 30 years of haptic inventions are buried in a fragmented literature that describes devices mechanically rather than by potential purpose. We conceived of Haptipedia to unlock this trove of examples: Haptipedia presents a device corpus for exploration through metadata that matter to both device and interaction designers. It is a taxonomy of device attributes that go beyond physical description to capture potential utility, applied to a growing database of 105 grounded force-feedback devices, and accessed through a public visualization that links utility to morphology. Haptipedia's design was driven by both systematic review of the haptic device literature and rich input from diverse haptic designers. We describe Haptipedia's reception (including hopes it will redefine device reporting standards) and our plans for its sustainability through community participation.

Critical to social human-robot interaction is a robot's emotional richness, expressed within the parameters of its physical display. While emotion arousal is straightforward to convey, human valence (positivity) evaluations are famously ambiguous, whether we are assessing other humans or a robot. Imagine someone breathing raggedly: are they nervous, or excited? To assess the premise that irregular breathing connotes low valence (emotion negativity), we implemented different levels of breathing variability and complexity in simple furry robots. We asked 10 participants to watch and feel the behaviors, rate their valence, and explain their impressions. While a quantitative exploration of new and previous data showed correlation between multi-scale entropy and valence, the rich narratives revealed by thematic analysis of participant explanations call into question whether a single motion can, alone, be unambiguously valenced. Based on this evidence that people perceive robots as having inner lives, we recommend ways to build up narrative contexts over multiple interactions.

When refining or personalizing a design, we count on being able to modify or move an element by changing its parameters rather than creating it anew in a different form or location—a standard utility in graphic and auditory authoring tools. Similarly, we need to tune vibrotactile sensations to fit new use cases, distinguish members of communicative icon sets, and personalize items. For tactile vibration display, however, we lack knowledge of the human perceptual mappings that must underlie such tools. Based on evidence that affective dimensions are a natural way to tune vibrations for practical purposes, we attempted to manipulate perception along three emotion dimensions (agitation,liveliness, andstrangeness) using engineering parameters of hypothesized relevance. Results from two user studies show that an automatable algorithm can increase a vibration’s perceivedagitationandlivelinessto different degrees via signal energy, while increasing its discontinuity or randomness makes it morestrange. These continuous mappings apply across diverse base vibrations; the extent of achievable emotion change varies. These results illustrate the potential for developing vibrotactile emotion controls as efficient tuning for designers and end-users.

Social robots must be believable to be effective; but creating believable, affectively expressive robot behaviours requires time and skill. Inspired by the directness with which performers use their voices to craft characters, we introduce Voodle (vocal doodling), which uses the form of utterances -- e.g., tone and rhythm -- to puppet and eventually control robot motion. Voodle offers an improvisational platform capable of conveying hard-to-express ideas like emotion. We created a working Voodle system by collecting a set of vocal features and associated robot motions, then incorporating them into a prototype for sketching robot behaviour. We explored and refined Voodle's expressive capacity by engaging expert performers in an iterative design process. We found that users develop a personal language with Voodle; that a vocalization's meaning changed with narrative context; and that voodling imparts a sense of life to the robot, inviting designers to suspend disbelief and engage in a playful, conversational style of design.

Haptic force feedback systems are unique in their ability to dynamically render physical representations. Although haptic devices have shown promise for supporting learning, prior work mainly describes results of haptic-supported learning without identifying underlying learning mechanisms. To this end, we designed a haptic-supported learning environment and analyzed four students who used it to make connections between two different mathematical representations of sine and cosine: the unit circle, and their graph on the Cartesian plane. We highlight moments where students made connections between the representations, and identify how the haptic feedback supported these moments of insight. We use this evidence in support of a proposed theoretical and design framework for educational haptics. This framework captures four types of haptic representations, and focuses on one -- the haptic bridge -- that effectively scaffolds sense-making with multiple representations.

Social robots that physically display emotion invite natural communication with their human interlocutors, enabling applications like robot-assisted therapy where a complex robot's breathing influences human emotional and physiological state. Using DIY fabrication and assembly, we explore how simple 1-DOF robots can express affect with economy and user customizability, leveraging open-source designs.

Touchscreen input in commercial aircraft cockpits offers potential advantages, including ease of use, modifiability, and reduced weight. However, tolerance to turbulence is a challenge for their deployment. To better understand the impact of turbulence on cockpit input methods we conducted a comparative study of user performance with three input methods -- touch, trackball (as currently used in commercial aircraft), and a touchscreen stencil overlay designed to assist finger stabilization. These input methods were compared across a variety of interactive tasks and at three levels of simulated turbulence (none, low, and high). Results showed that performance degrades and subjective workload increases as vibration increases. Touch-based interaction was faster than the trackball when precision requirements were low (at all vibrations), but it was slower and less accurate for more precise pointing, particularly at high vibrations. The stencil did not improve touch selection times, although it did reduce errors on small targets at high vibrations, but only when finger lift-off errors had been eliminated by a timeout. Our work provides new information on the types of tasks affected by turbulence and the input mechanisms that perform best under different levels of vibration.

Today's advances in tactile sensing, wearable, situated and context-aware computing and robotics are spurring new ideas about how to configure touch-centered interactions in terms of roles and utility, which in turn expose new technical and social design questions. But while haptic actuation and sensing technology is improving, incorporating them into a real-world design process still brings many challenges. In this talk I'll focus on how my group has approached research both into viable roles and design languages for physical communication, and of what is needed to support their design. I'll explore this in the context of several examples, such as translating users' cognitive frameworks for processing tactile signals into guidelines and tools to create learnable message vocabularies; exploiting low-cost, stretchy touch sensors and machine learning touch recognition to raise the 'emotional intelligence' of social human-robot interaction through bidirectional communication; and how such sensing and simple outputs can transform other interactions that are situated in the physical world rather than on a traditional computing device.

Vibrotactile (VT) display is becoming a standard component of informative user experience, where notifications and feedback must convey information eyes-free. However, effective design is hindered by incomplete understanding of relevant perceptual qualities, together with the need for user feedback to be accessed in-situ. To access evaluation streamlining now common in visual design, we introduce proxy modalities as a way to crowdsource VT sensations by reliably communicating high-level features through a crowd-accessible channel. We investigate two proxy modalities to represent a high-fidelity tactor: a new VT visualization, and low-fidelity vibratory translations playable on commodity smartphones. We translated 10 high-fidelity vibrations into both modalities, and in two user studies found that both proxy modalities can communicate affective features, and are consistent when deployed remotely over Mechanical Turk. We analyze fit of features to modalities, and suggest future improvements.

With advances in sensor and actuator design, intelligent computing techniques and personal care robotics, today's robots hold promise as fully interactive, therapeutic human companions. To achieve this ambitious goal, key interaction components must be identified and then systematically designed and evaluated. Based on successes of human-animal therapy, we propose affective touch as one such component. Delivering this adjunct in a controllable robot form allows us to examine its efficacy for therapeutic applications such as anxiety management. With an approach grounded in social cognitive theories for human-animal relations, we deployed a social robot, the Haptic Creature, in an interaction designed to be calming: participants held the robot on their laps and stroked it as it was breathing. As a result, their heart and respiration rates significantly decreased relative to stroking a non-breathing robot. They also reported themselves as calmer and happier.

With haptics now common in consumer devices, diversity in tactile perception and aesthetic preferences confound haptic designers. End-user customization out of example sets is an obvious solution, but haptic collections are notoriously difficult to explore. This work addresses the provision of easy and highly navigable access to large, diverse sets of vibrotactile stimuli, on the premise that multiple access pathways facilitate discovery and engagement. We propose and examine five disparate organization schemes (taxonomies), describe how we created a 120-item library with diverse functional and affective characteristics, and present VibViz, an interactive tool for end-user library navigation and our own investigation of how different taxonomies can assist navigation. An exploratory user study with and of VibViz suggests that most users gravitate towards an organization based on sensory and emotional terms, but also exposes rich variations in their navigation patterns and insights into the basis of effective haptic library navigation.

We present a real-time touch gesture recognition system using a low-cost fabric pressure sensor mounted on a small zoomorphic robot, affectionately called the `CuddleBit'. We explore the relationship between gesture recognition and affect through the lens of human-robot interaction. We demonstrate our real-time gesture recognition system, including both software and hardware, and a haptic display that brings the CuddleBit to life.

Social touch is an essential non-verbal channel whose great interactive potential can be realized by the ability to recognize gestures performed on inviting surfaces. To assess impact on recognition performance of sensor motion, substrate and coverings, we collected gesture data from a low-cost multitouch fabric pressure-location sensor while varying these factors. For six gestures most relevant in a haptic social robot context plus a no-touch control, we conducted two studies, with the sensor (1) stationary, varying substrate and cover (n=10); and (2) attached to a robot under a fur covering, flexing or stationary (n=16). For a stationary sensor, a random forest model achieved 90.0% recognition accuracy (chance 14.2%) when trained on all data, but as high as 94.6% (mean 89.1%) when trained on the same individual. A curved, flexing surface achieved 79.4% overall but averaged 85.7% when trained and tested on the same individual. These results suggest that under realistic conditions, recognition with this type of flexible sensor is sufficient for many applications of interactive social touch. We further found evidence that users exhibit an idiosyncratic `touch signature', with potential to identify the toucher. Both findings enable varied contexts of affective or functional touch communication, from physically interactive robots to any touch-sensitive object.

Advances in the field of touch recognition could open up applications for touch-based interaction in areas such as Human-Robot Interaction (HRI). We extended this challenge to the research community working on multimodal interaction with the goal of sparking interest in the touch modality and to promote exploration of the use of data processing techniques from other more mature modalities for touch recognition. Two data sets were made available containing labeled pressure sensor data of social touch gestures that were performed by touching a touch-sensitive surface with the hand. Each set was collected from similar sensor grids, but under conditions reflecting different application orientations: CoST: Corpus of Social Touch and HAART: The Human-Animal Affective Robot Touch gesture set. In this paper we describe the challenge protocol and summarize the results from the touch challenge hosted in conjunction with the 2015 ACM International Conference on Multimodal Interaction (ICMI). The most important outcomes of the challenges were: (1) transferring techniques from other modalities, such as image processing, speech, and human action recognition provided valuable feature sets; (2) gesture classification confusions were similar despite the various data processing methods used.

Chairs, wearables, and handhelds have become popular sites for spatial tactile display. Visual animators, already expert in using time and space to portray motion, could readily transfer their skills to produce rich haptic sensations if given the right tools. We introduce the tactile animation object, a directly manipulated phantom tactile sensation. This abstraction has two key benefits: 1) efficient, creative, iterative control of spatiotemporal sensations, and 2) the potential to support a variety of tactile grids, including sparse displays. We present Mango, an editing tool for animators, including its rendering pipeline and perceptually-optimized interpolation algorithm for sparse vibrotactile grids. In our evaluation, professional animators found it easy to create a variety of vibrotactile patterns, with both experts and novices preferring the tactile animation object over controlling actuators individually.

In this paper, we compare a touch-based human-to-robot command scheme with traditional button commands in a series of human-robot collaborative assembly tasks. We find a mapping between command style and task outcome that depends on task complexity and is influenced by robot feel. In our direct touch-based scheme, the user commands the robot through direct physical contact by tapping and pushing the robot. With a small, compliant desktop robot and a simple, scripted, bolt insertion task, button commands performed slightly better than direct physical commands in quantitative task performance metrics and qualitative user preference. In a second study with a human-scale, stiffer robot arm, physical commands performed better than button commands in a more complex and less scripted bolt insertion task, which greatly outperformed using buttons in a cooperative positioning task. We conclude that commanding a robot through direct force-transmitting contact can decrease task completion time, aid in teamwork, and improve user experience in appropriately chosen tasks. We achieve our haptic commands using only robot position sensors, demonstrating that direct, intuitive physical command is an option for existing position-controlled industrial robots.

Coming generations of robots will share physical space with humans, engaging in contact interactions (physical Human Robot Interaction, or pHRI) as they carry out cooperative tasks. This special issue turns a spotlight on the specific roles that crafted haptic interaction can play in cooperation and communication between a human and a robotic partner, from the viewpoints of human needs, capabilities, and expectations and of engineering implementation.

To make a single diagnosis, today's radiologists must examine thousands of images; yet little effort has been put into refining this time-consuming, repetitive task. Meanwhile, automatic or radiologist-generated annotations may impact how radiologists navigate image stacks as they review lesions of interest. Observation and/or interviews of 19 radiologists revealed that stack scrolling dominated the resulting task examples. We iteratively crafted and obtained radiologist feedback for a variety of prototypes, then evaluated their scrolling and annotation-review support for lay users. With a simplified stack seeded with correct / incorrect annotations, we compared the effect of four scrolling techniques (traditional scrollwheel and click-and-drag, plus sliding-touch, and tilt rate control) and visual vs. haptic annotation cues on scrolling dynamics, detection accuracy and subjective factors. Scrollwheel was fastest overall, and combined visual / haptic annotation cues sped target-finding relative to either modality alone. We share insights on integrating our findings into radiologist practice.

In this paper we provide empirical evidence that using humanlike gaze cues during human-robot handovers can improve the timing and perceived quality of the handover event. Handovers serve as the foundation of many human-robot tasks. Fluent, legible handover interactions require appropriate nonverbal cues to signal handover intent, location and timing. Inspired by observations of human-human handovers, we implemented gaze behaviors on a PR2 humanoid robot. The robot handed over water bottles to a total of 102 naïve subjects while varying its gaze behaviour: no gaze, gaze designed to elicit shared attention at the handover location, and the shared attention gaze complemented with a turn-taking cue. We compared subject perception of and reaction time to the robot-initiated handovers across the three gaze conditions. Results indicate that subjects reach for the offered object significantly earlier when a robot provides a shared attention gaze cue during a handover. We also observed a statistical trend of subjects preferring handovers with turn-taking gaze cues over the other conditions. Our work demonstrates that gaze can play a key role in improving user experience of human-robot handovers, and help make handovers fast and fluent.

To moderate oral presentations a chair must manage time, and communicate time parameters to speakers through a variety of means. But speakers often miss time cues, chairs cannot confirm their receipt, and the broken dialogue can be a sideshow for the audience. We developed HaNS, a wireless wrist-worn chair-speaker Haptic Notification System that delivers tactile cues for time-managing oral presentations, and performed field observations at university research seminars and two mid-sized academic conferences (input from 66 speakers, 21 chairs, and 65 audience members). Results indicate that HaNS can improve a user's awareness of time, facilitate chair-speaker coordination, and reduce distraction of speaker and audience through its private communication channel. Eliminating overruns will require improvement in speaker 'internal' control, which our results suggest HaNS can also support given practice. We conclude with design guidelines for both conference-deployed and personal timing tools, using touch or another notification modality.

Affective response may dominate users' reactions to the synthesized tactile sensations that are proliferating in today's handheld and gaming devices, yet it is largely unmeasured, modeled or characterized. A better understanding of user perception will aid the design of tactile behavior that engages touch, with an experience that satisfies rather than intrudes. We measured 30 subjects' affective response to vibrations varying in rhythm and frequency, then examined how differences in demographic, everyday use of touch, and tactile processing abilities contribute to variations in affective response. To this end, we developed five affective and sensory rating scales and two tactile performance tasks, and also employed a published `Need for Touch' (NFT) questionnaire. Subjects' ratings, aggregated, showed significant correlations among the five scales and significant effect of the signal content (rhythm and frequency). Ratings varied considerably among subjects, but this variation did not coincide with demographic, NFT score or tactile task performance. The linkages found among the rating scales confirm this as a promising approach. The next step towards a comprehensive picture of individuals' patterns of affective response to tactile sensations entails pruning, integration and redundancy reduction of these scales, then their formal validation.

New developments, innovations, and advancements in robotic technology are paving the way for intelligent robots to enable, support, and enhance the capabilities of human workers in manufacturing environments. We envision future industrial robot assistants that support workers in their tasks, advancing manufacturing quality and processes and increasing productivity. However, this requires new channels of fine-grained, fast and reliable communication. In this research we examined the communication required for human-robot collaboration in a vehicle door assembly scenario. We identified potential communicative gestures applicable to this scenario, implemented these gestures on a Barrett WAMTM1manipulator, and evaluated them in terms of human recognition rate and response time in a real-time interaction. Response time analysis reveals insights into the communicative structure of robot motions; namely, key short gesture segments include the bulk of the communicative information. These results will help us design more efficient and fluid task flow in human-robot interaction scenarios.

Persuasive technology is now mobile and context-aware. Intelligent analysis of accelerometer signals in smartphones and other specialized devices has recently been used to classify activity (e.g., distinguishing walking from cycling) to encourage physical activity, sustainable transport, and other social goals. Unfortunately, results vary drastically due to differences in methodology and problem domain. The present report begins by structuring a survey of current work within a new framework, which highlights comparable characteristics between studies; this provided a tool by which we and others can understand the current state-of-the art and guide research towards existing gaps. We then present a new user study, positioned in an identified gap, that pushes limits of current success with a challenging problem: the real-time classification of 15 similar and novel gaits suitable for several persuasive application areas, focused on the growing phenomenon of exercise games. We achieve a mean correct classification rate of 78.1% of all 15 gaits with a minimal amount of personalized training of the classifier for each participant when carried in any of 6 different carrying locations (not known a priori). When narrowed to a subset of four gaits and one location that is known, this improves to means of 92.2% with and 87.2% without personalization. Finally, we group our findings into design guidelines and quantify variation in accuracy when an algorithm is trained for a known location and participant.

Over the last decade, the surprising fact has emerged that machines can possess therapeutic power. Due to the many healing qualities of touch, one route to such power is through haptic emotional interaction, which requires sophisticated touch sensing and interpretation. We explore the development of touch recognition technologies in the context of a furry artificial lap-pet, with the ultimate goal of creating therapeutic interactions by sensing human emotion through touch. In this work, we build upon a previous design for a new type of fur-based touch sensor. Here, we integrate our fur sensor with a piezoresistive fabric location/pressure sensor, and adapt the combined design to cover a curved creature-like object. We then use this interface to collect synchronized time-series data from the two sensors, and perform machine learning analysis to recognize 9 key affective touch gestures. In a study of 16 participants, our model averages 94% recognition accuracy when trained on individuals, and 86% when applied to the combined set of all participants. The model can also recognize which participant is touching the prototype with 79% accuracy. These results promise a new generation of emotionally intelligent machines, enabled by affective touch gesture recognition.

In this paper we explore the potential and limitations of vibrotactile displays in practical wearable applications, by comparing users' detection rate and response time to stimuli applied across the body in varied conditions. We examined which body locations are more sensitive to vibrations and more affected by movement; whether visual workload, expectation of location, or gender impact performance; and if users have subjective preferences to any of these conditions. In two experiments we compared these factors using five vibration intensities on up to 13 body locations. Our contributions are comparisons of tactile detection performance under conditions typifying mobile use, an experiment design that supports further investigation in vibrotactile communication, and guidelines for optimal display location given intended use.

Touch interactions have refreshed some of the 'glowing enthusiasm' of thirty years ago for direct manipulation interfaces. However, today's touch technologies, whose interactions are supported by graphics, sounds or crude clicks, have a tactile sameness and gaps in usability. We use a Large Area Tactile Pattern Display (LATPaD) to examine design possibilities and outcomes when touch interactions are enhanced with variable surface friction. In a series of four studies, we first confirm that variable friction gives significant performance advantages in low-level targeting activities. We then explore the design space of variable friction interface controls and assess user reactions. Most importantly, we demonstrate that variable friction can have a positive impact on the enjoyment, engagement and sense of realism experienced by users of touch interfaces.

This work explores a novel interaction paradigm driven by implicit, low-attention user control, accomplished by monitoring a user's physiological state. We have designed and prototyped this interaction for a first use case of bookmarking an audio stream, to holistically explore the implicit interaction concept. Here, a user's galvanic skin conductance (GSR) is monitored for orienting responses (ORs) to external interruptions; our prototype automatically bookmarks the media such that the user can attend to the interruption, then resume listening from the point he/she is interrupted. To test this approach's viability, we addressed questions such as: does GSR exhibit a detectable response to interruptions, and how should the interaction utilize this information? In evaluating this system in a controlled environment, we found an OR detection accuracy of 84%; users provided subjective feedback on its accuracy and utility.

A haptic icon is a short physical stimulus attached to a simple meaning, which provides information and feedback to a user. To scale the utility demonstrated for small icon sets to larger ones, we need efficient strategies to help users learn subtle distinctions among stimuli, in a modality for which they may not hold detailed descriptive percepts. This paper investigates the effect of naming haptic stimuli - i.e. explicitly creating a linguistic marker - on the accuracy with which users are able to identify, distinguish, and recall stimuli. We conducted a between-subjects experiment using 60 participants equally divided among three naming conditions: no names, pre-selected non-descriptive names, and self-selected names. The experiment examined the impact of naming strategy on the ability of participants to identify stimuli in a nonverbal matching test, and on remembering stimulus names. For this challenging task and the degree of learning afforded, naming did not significantly impact accuracy of matching stimuli to meanings for all participants. However, more than twice of many of those allowed to choose names reported the ability to remember and distinguish stimuli than those required to use non-descriptive names, and many participants felt that the names were useful. Of middle-performing participants, the self-selected names group performed significantly better than the non-descriptive names group, and appeared to progress more quickly in learning. We summarize evidence for a trend that might widen with refined naming strategies and more extensive learning.

The vibration behavior of acoustic music instruments, as perceived through touch, is known to play an important role in the interaction between performers and music instruments. This research explores the relevance and utility of including a tactile simulation of this behavior in computer music interfaces, in a snapshot taken during early learning, and including tradeoffs of using this channel for other information. Vibrotactile (VT) renderings of different types of sounds were presented to subjects via a handle fitted with vibration actuators. Subjects (1) evaluated consistency of the sound with the VT rendering, (2) identified from a “lineup” the VT rendering that did not match a sound, and (3) provided data on how inclusion of VT cues in a rhythm-tapping task affected perception quality of the VT rendering. For the last, a distractor task allowed us to measure both the usability of the cues and the degradation in perception of the VT rendering. All renderings were either played directly through vibration transducers, or first altered by modifying the sound's frequency content. VT playback was delivered at intensities similar to those experienced in traditional acoustic music instruments. Subjects indicated that specific alterations to the original, direct VT rendering were consistent with the source sound more often than the original was, and were able to differentiate between correct and incorrect renderings of a source sound. However, the latter ability was masked when subjects were provided with and able to effectively utilize extra VT cues added to the VT feedback to improve their performance, suggesting that the VT cues were of greater utility than the VT mimicking. We discuss the relevance of these findings on the design of computer music interfaces.

The Haptic Creature is a small, animal-like robot we have developed to investigate the role of touch in communicating emotions between humans and robots. This paper presents a study examining how successful our robot is at communicating its emotional state through touch. Results show that, regardless of the human's gender or background with animals, the robot is effective in communicating its state of arousal but less so for valence. Also included are descriptions of the design of the Haptic Creature's emotion model and suggested improvements based on results of the study.

Physical controls such as knobs, sliders, and buttons are experiencing a revival as many computing systems progress from personal computing architectures towards ubiquitous computing architectures. We demonstrate a process for measuring and comparing visceral emotional responses of a physical control to performance results of a target acquisition task. In our user study, participants experienced mechanical and rendered friction, inertia, and detent dynamics as they turned a haptic knob towards graphical targets of two different widths and amplitudes. Together, this process and user study provide novel affect- and performance-based design guidance to developers of physical controls for emerging ubiquitous computing environments. Our work bridges extensive human factors work in mechanical systems that peaked in the 1960's, to contemporary trends, with a goal of integrating mechatronic controls into emerging ubiquitous computing systems.

Adding expressive haptic feedback to mobile devices has great potential to improve their usability, particularly in multitasking situations where one's visual attention is required. Piezoelectric actuators are emerging as one suitable technology for rendering expressive haptic feedback on mobile devices. We describe the design of redundant piezoelectric haptic augmentations of touchscreen GUI buttons, progress bars, and scroll bars, and their evaluation under varying cognitive load. Our haptically augmented progress bars and scroll bars led to significantly faster task completion, and favourable subjective reactions. We further discuss resulting insights into designing useful haptic feedback for touchscreens and highlight challenges, including means of enhancing usability, types of interactions where value is maximized, difficulty in disambiguating background from foreground signals, tradeoffs in haptic strength vs. resolution, and subtleties in evaluating these types of interactions.

This paper describes the design, construction, and initial evaluation of a handheld information device that supports combined tactile and graphical interaction. The design comprises a liquid crystal graphic display co-located with a miniature, low-power, distributed tactile transducer. This transducer can create electronically-controlled lateral skin deformation patterns which give the sensation of sliding over small shapes. It is integrated within a slider mechanism to control scrolling. It also functions as a detent when pushing on it. Tactile feedback and the combination of visual and tactile feedback in a mobile context enable the development of new functions, such as multimodal navigation within large graphic spaces

Mobile interaction can potentially be enhanced with well-designed haptic control and display. However, advances have been limited by a vicious cycle whereby inadequate haptic technology obstructs inception of vitalizing applications. We present the first stages of a systematic design effort to break that cycle, beginning with specific usage scenarios and a new handheld display platform based on lateral skin stretch. Results of a perceptual device characterization inform mappings between device capabilities and specific roles in mobile interaction, and the next step of hardware re-engineering.

A haptic phoneme represents the smallest unit of a constructed haptic signal to which a meaning can be assigned. These haptic phonemes can be combined serially or in parallel to form haptic words, or haptic icons, which can hold more elaborate meanings for their users. Here, we use phonemes which consist of brief (<2 seconds) haptic stimuli composed of a simple waveform at a constant frequency and amplitude. Building on previous results showing that a set of 12 such haptic stimuli can be perceptually distinguished, here we test learnability and recall of associations for arbitrarily chosen stimulus-meaning pairs. We found that users could consistently recall an arbitrary association between a haptic stimulus and its assigned arbitrary meaning in a 9-phoneme set, during a 45 minute test period following a reinforced learning stage.

Intelligent systems are increasingly able to offer real-time information relevant to a user's manual control of an interactive system, such as dynamic system control space constraints for animation control and driving. However, it is difficult to present this information in a usable manner and other approaches which have employed haptic cues for manual control in "slow" systems often lead to instabilities in highly dynamic tasks. We present a predictive haptic guidance method based on a look-ahead algorithm, along with a user evaluation which compares it with other approaches (no guidance and a standard potential-field method) in a 1-DoF steered path-following scenario. Look-ahead guidance outperformed the other methods in both quantitative performance and subjective preference across a range of path complexity and visibility and a force analysis demonstrated that it applied smaller and fewer forces to users. These results (which appear to derive from the predictive guidance's supporting users in taking earlier and more subtle corrective action) suggest the potential of predictive methods in aiding manual control of dynamic interactive tasks where intelligent support is available.

Current methods of playlist creation and maintenance do not support user needs, especially in a mobile context. Furthermore, they do not scale: studies show that users with large mp3 collections have abandoned the concept of playlists. To remedy the usability problems associated with playlist creation and navigation - in particular, reliance on visual feedback and the absence of rapid content scanning mechanisms - we propose a system that utilizes the haptic channel. A necessary first step in this objective is the creation of a haptic mapping for music. In this paper, we describe an exploratory study addressed at understanding the feasibility, with respect to learnability and usability, of efficient, eyes-free playlist navigation based on symbolic haptic renderings of key song parameters. Users were able to learn haptic mappings for music parameters to usable accuracy with 4 minutes of training. These results indicate promise for the approach and support for continued effort in both improving the rendering scheme and implementing the haptic playlist system.

Visual information overload is a threat to the interpretation of displays presenting large data sets or complex application environments. To combat this problem, researchers have begun to explore how haptic feedback can be used as another means for information transmission. In this paper, we show that people can perceive and accurately process haptically rendered ordinal data while under cognitive workload. We evaluate three haptic models for rendering ordinal data with participants who were performing a taxing visual tracking task. The evaluation demonstrates that information rendered by these models is perceptually available even when users are visually busy. This preliminary research has promising implications for haptic augmentation of visual displays for information visualization.

This paper describes the Hapticat, a device we developed to study affect through touch. Though intentionally not highly zoomorphic, the device borrows behaviors from pets and the rich manner in which they haptically communicate with humans. The Hapticat has four degrees of freedom to express itself: a pair of ear-like appendages, a breathing mechanism, a purring mechanism, and a warming element. Combinations of levels for these controls are used to define the five active haptic responses: playing dead, asleep, content, happy, and upset,. In the paper we present the design considerations and implementation details of the device. We also detail a preliminary observational study where participants interacted with the Hapticat through touch. To compare the effects of haptic feedback, the device presented either active haptic renderings or none at all. Participants reported which of the five responses they believed the Hapticat rendered, as well as their degree of affect to the device. We observed that participants' expectations of the device's response to various haptic stimuli correlated with our mappings. We also observed that participants were able to reasonably recognize three of the five response renderings, while having difficulty discriminating between happy and content states. Finally, we found that participants registered a broader range of affect when active haptic renderings were applied as compared to when none were presented.

In this paper we consider the general goal of supporting physical manipulation of digital audio in a specific context: the performance disk jockey (DJ) seeking to migrate from vinyl to digital media. We classify both the DJ's traditional processes and tools and the field's newest technology.D'Groove, our own technological contribution, is a force feedback turntable used to manipulate digital audio in novel ways. We present an observational study of professional DJ's using D'Groove, and discuss this approach's attributes and directions for future augmentation. Finally, we extend our conclusions about the DJ's emerging needs to the broader domain of digital audio manipulation.

This work describes a platform devised to explore the impact on task execution in a virtual environment of the quality, and therefore cost, of the system's haptic hardware. This platform is a complex haptic interface in which hardware quality can be varied in simulation. Software intercepts the position and force signals between the haptic hardware and the virtual environment software, and alters them to supply the effect of increased friction, cogging, backlash, inertia and/or lower force output. All parameters of the introduced effects can be set independently or in combination and on a continuous scale; a primary contribution is the creation of haptically realistic effect models that are stable in combination on complex hardware. This work is part of a larger project in which we test the effect of the simulated degradations on surgeons performance in simulated laparoscopic tasks.

We introduce a set of techniques for haptically manipulating digital media such as video, audio, voicemail and computer graphics, utilizing virtual mediating dynamic models based on intuitive physical metaphors. For example, a video sequence can be modeled by linking its motion to a heavy spinning virtual wheel: the user browses by grasping a physical force-feedback knob and engaging the virtual wheel through a simulated clutch to spin or brake it, while feeling the passage of individual frames. These systems were implemented on a collection of single axis actuated displays (knobs and sliders), equipped with orthogonal force sensing to enhance their expressive potential. We demonstrate how continuous interaction through a haptically actuated device rather than discrete button and key presses can produce simple yet powerful tools that leverage physical intuition.

Discrete and continuous modes of manual control are fundamentally different: buttons select or change state, while handles persistently modulate an analog parameter. User interfaces for many electronically aided tasks afford only one of these modes when both are needed. We describe an integration of two kinds of physical interfaces (tagged objects and force feedback) that enables seamless execution of such multimodal tasks while applying the benefits of physicality; and demonstrate application scenarios with conceptual and engineering prototypes. Our emphasis is on sharing insights gained in a design case study, including expert user reactions.

Haptic feedback is a design element for human-computer interfaces, and this paper discusses when and how it can be used to best effect in interactive applications. It begins with consideration of the unique attributes of the touch sense in physiological and psychological terms, and the nature of information and control that touching provides. It reviews where active touching helps, by setting forth the forms it may take and important parameters that describe it; and evaluates the specific benefits it offers to contemporary interface problems. It ends with a proposal for a simple interaction model that emphasizes holistic design principles, and highlights issues that arise in the process of creating specific haptic interfaces.