Summary Wearable Cognitive Assistance (WCA) amplifies human cognition in real time through a wearable device and low‐latency wireless access to edge computing infrastructure. It is inspired by, and broadens, the metaphor of GPS navigation tools that provide real‐time step‐by‐step guidance, with prompt error detection and correction. WCA applications are likely to be transformative in education, health care, industrial troubleshooting, manufacturing, assisted driving, and sports training. Today, WCA application development is difficult and slow, requiring skills in areas such as machine learning and computer vision that are not widespread among software developers. This paper describesAjalon,an authoring toolchain for WCA applications that reduces the skill and effort needed at each step of the development pipeline. Our evaluation shows that Ajalon significantly reduces the effort needed to create new WCA applications.

We introduce a novel, perceptually derived metric (P - Reverb) that relates the just-noticeable difference (JND) of the early sound field (also called early reflections) to the late sound field (known as late reflections or reverberation). Early and late reflections are crucial components of the sound field and provide multiple perceptual cues for auditory displays. We conduct two extensive user evaluations that relate the JNDs of early reflections and late reverberation in terms of the mean-free path of the environment and present a novel P - Reverb metric. Our metric is used to estimate dynamic reverberation characteristics efficiently in terms of important parameters like reverberation time (RT60). We show the numerical accuracy of our P - Reverb metric in estimating RT60. Finally, we use our metric to design an interactive sound propagation algorithm and demonstrate its effectiveness on various benchmarks.

Modal sound synthesis has been used to create realistic sounds from rigid-body objects, but requires accurate real-world material parameters. These material parameters can be estimated from recorded sounds of an impacted object, but external factors can interfere with accurate parameter estimation. We present a novel technique for estimating the damping parameters of materials from recorded impact sounds that probabilistically models these external factors. We represent the combined effects of material damping, support damping, and sampling inaccuracies with a probabilistic generative model, then use maximum likelihood estimation to fit a damping model to recorded data. This technique greatly reduces the human effort needed and does not require the precise object geometry or the exact hit location. We validate the effectiveness of this technique with a comprehensive analysis of a synthetic dataset and a perceptual study on object identification. We also present a study establishing human performance on the same parameter estimation task for comparison.

Immersive experiences seek to engage the full sensory system in ways that words, pictures, or touch alone cannot. With respect to the haptic system, however, physical feedback has been provided primarily with handheld tactile experiences or vibration-based designs, largely ignoring both pressure receptors and the full upper-body area as conduits for expressing meaning that is consistent with sight and sound. We extend the potential for immersion along these dimensions with the Force Jacket, a novel array of pneumatically-actuated airbags and force sensors that provide precisely directed force and high frequency vibrations to the upper body. We describe the pneumatic hardware and force control algorithms, user studies to verify perception of airbag location and pressure magnitude, and subsequent studies to define full-torso, pressure and vibration-based feel effects such as punch, hug, and snake moving across the body. We also discuss the use of those effects in prototype virtual reality applications.

No abstract available.

As sound propagation algorithms become faster and more accurate, the question arises as to whether the additional efforts to improve fidelity actually offer perceptual benefits over existing techniques. Could environmental sound effects go the way of music, where lower-fidelity compressed versions are actually favored by listeners? Here we address this issue with two acoustic phenomena that are known to have perceptual effects on humans and that, accordingly, might be expected to heighten their experience with simulated environments. We present two studies comparing listeners’ perceptual response to both accurate and approximate algorithms simulating two key acoustic effects: diffraction and reverberation. For each effect, we evaluate whether increased numerical accuracy of a propagation algorithm translates into increased perceptual differentiation in interactive virtual environments. Our results suggest that auditory perception does benefit from the increased accuracy, with subjects showing better perceptual differentiation when experiencing the more accurate rendering method: the diffraction experiment shows a more linearly decaying sound field (with respect to the diffraction angle) for the accurate diffraction method, whereas the reverberation experiment shows that more accurate reverberation, after modest user experience, results in near-logarithmic response to increasing room volume.

Research on children's cognitive development has demonstrated the positive effects of listening to stories. However, traditional story listening is losing its appeal to other entertainment technology such as video games. Hence, there is growing interest in studying the influence of ancillary media such as sound and interactive effects, although haptic sensory input has remained relatively unexploited. We implemented a haptic vest that generates vibrotactile stimulation related to story content to augment story listening. Study 1 showed that 5- and 6-year olds, but not 4-year olds, could associate haptic effects with semantic interpretations. In Study 2, children listened to stories containing elements with or without haptic effects. The 5- and 6-year olds showed better comprehension of the haptically-signaled content in the higher-performance story. The results provide initial evidence that haptic effects can potentially enhance the reading/listening experience of children beyond 4 years.

Many have experienced vastness, the feeling when the visual space seems to extend without limits away from you, making you feel like a small element within the space. For over 200 years, people have been writing about this experience, for example stating that vastness is important to the experience of awe [Mikulak 2015]. Yet vastness has received little attention in empirical research. Specifically, it is unknown which aspects of the visual stimulus contribute to perceived vastness. This may be due to the inherent difficulties in presenting a variety of vast stimuli while varying only specific visual cues. Using virtual reality addresses these difficulties, as this technology provides precise control over the presented visual stimuli. Here we investigate whether the feeling of vastness can be evoked using virtual reality and explore potential objective measures to assess vastness. We used three different measures during this experiment: 1) An avatar height adjustment task where participants had to adjust an avatar to be equivalent to their own height as viewed from a distance, 2) a distance estimation task and 3) a subjective vastness rating task. These tasks were performed in four environments: a plain (used in all subsequent environments for the ground and sky surfaces), a forest, a mountain and the mountain and forest environments combined. Our results indicate that the feeling of vastness can indeed be experienced to various degrees in virtual environments, demonstrating the potential of VR as a tool for exploring the perception of vastness. Yet the results combined suggest that the percept of vastness is a rather complex construct.

Engaging children with traditional approaches in education, especially reading, grows ever more difficult in the face of their attachment to tablets and computer games. We explore the possibility of making the story reading experience more interesting and memorable for children using haptic augmentation. In this paper, we present FeelSleeve, an interface that allows children to feel story events in their hands while they are reading on a mobile device. FeelSleeve uses transducers and audio output from the tablet within a gloved attachment to create vibratory effects that are meaningfully related to story content. We describe a study investigating whether embedding such haptic feedback into stories enhances reading for six to nine year olds. Our results indicate that story events accompanied by haptic feedback are better comprehended and appear to be more salient in memory. These results provide evidence that haptic effects have the potential to improve children's reading experience and make it more memorable.

Handheld and wearable devices frequently engage users with simple haptic feedback, such as alerting, shaking, and pulsating. Here we explored intermanual apparent tactile motion-illusory movement between two hands-as a means to enrich such feedback. A series of psychophysical experiments determined the control space for generating smooth and consistent motion across the hands while users held the device. Experiment 1 calibrated the system and showed that vibrotactile detection thresholds decreased with increasing frequency, with similar trends for both hands. Experiment 2 measured effects of vibrotactile parameters on perceived motion. Both duration and temporal separation of stimuli, but not frequency and amplitude, affected subjective motion ratings. In Experiment 3, subjective ratings showed that stimuli with gradual onsets produced a stronger percept of motion than those with abrupt onsets. Finally, Experiment 4 determined a multimodal factor to match moving visual cues across the screen to moving tactile motion across hands. Our results showed compression of visual duration by the tactile system by a factor of approximately 1/3 at two test frequencies. The results of this research are useful for media designers and developers to generate reliable motion across the hands and integrate haptic motion with visual media.

Despite a long history of use in communication, haptic feedback is a relatively new addition to the toolbox of special effects. Unlike artists who use sound or vision, haptic designers cannot simply access libraries of effects that map cleanly to media content, and they lack even guiding principles for creating such effects. In this article, we make progress toward both capabilities: we generate a foundational library of usable haptic vocabulary and do so with a methodology that allows ongoing additions to the library in a principled and effective way. We define a feel effect as an explicit pairing between a meaningful linguistic phrase and a rendered haptic pattern. Our initial experiment demonstrates that users who have only their intrinsic language capacities, and no haptic expertise, can generate a core set of feel effects that lend themselves via semantic inference to the design of additional effects. The resulting collection of more than 40 effects covers a wide range of situations (including precipitation, animal locomotion, striking, and pulsating events) and is empirically shown to produce the named sensation for the majority of our test users in a second experiment. Our experiments demonstrate a unique and systematic approach to designing a vocabulary of haptic sensations that are related in both the semantic and parametric spaces.

This work investigates how haptic percepts are combined across two fingertips. Two single-degree-of-freedom haptic interfaces were used to present virtual bumps to the thumb and index finger of subjects' right hand. As subjects slid the two interfaces from left to right while maintaining a fixed finger separation, they would encounter one bump with the index finger and one with the thumb. The objective bump locations were varied randomly, from spatially coincident to separated by slightly more than the fingertips. Subjects were asked to report the number of bumps in the objective world and the number of times they encountered each bump, and also to point to the locations of the bumps. We found that subjects exhibited a strong bias toward reporting a single objective bump in the virtual world. However, the percept varied from one bump encountered twice, when the two virtual bumps were nearly spatially coincident, to one bump encountered once, which occurred when the two virtual bumps were close to finger separation and therefore encountered nearly simultaneously. The latter result is evidence for a kinesthetic grouping phenomenon: temporally synchronized sensations at multiple fingers can be perceptually collapsed into a single percept in both time and space.

We describe an experimental system for the evaluation of teleoperation performance. The system was used in two experiments where operators were assigned the task of discovering a buried object while minimizing contact forces. The studies i) demonstrated the advantage of haptic feedback in the discovery task, and ii) compared different methods for haptic feedback, including our new bimanual method, Touch and Guide in Tandem (TAGIT). Results show that TAGIT enables the effective workspace of the teleoperator to be expanded while minimizing forces from exploration and contact, reducing their variability, and reducing task completion times.

Perceiving the material properties of objects through touch is generally superior to the perception of shape. We review major material properties accessible through haptic interaction, along with theoretical accounts of the underlying perceptual processes. These include roughness, friction, compliance, and thermal properties. Subsequently, we describe algorithms that have been used to render these same material properties on haptic devices. We then point to applications that have capitalized on the accessibility of material through touch, including tactile displays, simulation of mechanical mechanisms in the automobile, and medical training simulators.

Accurately modeling the intrinsic material-dependent damping property for interactive sound rendering is a challenging problem. The Rayleigh damping model is commonly regarded as an adequate engineering model for interactive sound synthesis in virtual environment applications, but this assumption has never been rigorously analyzed. In this paper, we conduct a formal evaluation of this model. Our goal is to determine if auditory perception of material under Rayleigh damping assumption is 'geometry-invariant', i.e. if this approximation model is transferable across different shapes and sizes. First, audio recordings of same-material objects in various shapes and sizes are analyzed to determine if they can be approximated by the Rayleigh damping model with a single set of parameters. Next, we design and conduct a series of psychoacoustic experiments, in subjects evaluate if audio clips synthesized using the Rayleigh damping model are from the same material, when we alter the material, shape, and size parameters. Through both quantitative and qualitative evaluation, we show that the acoustic properties of the Rayleigh damping model for a single material is generally preserved across different geometries of objects consisting of homogeneous materials and is therefore a suitable, geometry-invariant sound model. Our study results also show that consistent with prior crossmodal expectations, visual perception of geometry can affect the auditory perception of materials. These findings facilitate the wide adoption of Rayleigh damping for interactive auditory systems and enable reuse of material parameters under this approximation model across different shapes and sizes, without laborious per-object parameter tuning.

"Psychophysics" is a an approach to evaluating human perception and action capabilities that emphasizes control over the stimulus environment. Virtual environments provide an ideal setting for psychophysical research, as they facilitate not only stimulus control but precise measurement of performance. In my research I have used the psychophysical approach to inform the design and evaluation of multi-modal interfaces that enable action in remote or virtual worlds or that compensate for sensory-motor impairment in the physical environment of the user. This talk will describe such projects, emphasizing the value of behavioral science to interface engineering.

Although it is reported in the literature that haptic feedback leads to improved performance in kinesthetic collaborative tasks, it has not been investigated so far whether this advantage is accompanied by a higher physical workload. This paper is an initial effort to examine efficiency in haptic interaction: We relate physical effort to a performance outcome in a virtual pursuit tracking task. An experimental study is conducted to compare efficiency in a collaborative mutual haptic feedback condition to three control conditions, where participants either acted alone or collaboratively without haptic feedback from the partner. Results show that reciprocal haptic feedback does not improve efficiency, although participants' performance was generally improved when doing the task with a partner, relative to executing it alone. This is due to the greater effort associated with physical connection between partners. However, the effort is more fairly distributed between partners when haptic feedback from the partner is provided. Haptic feedback may be more efficient when the amount of necessary communication between partners increases compared to the task studied here.

The present study examined the impact of augmented-reality visualization, in comparison to conventional ultrasound (CUS), on the learning of ultrasound-guided needle insertion. Whereas CUS requires cognitive processes for localizing targets, our augmented-reality device, called the “sonic flashlight” (SF) enables direct perceptual guidance. Participants guided a needle to an ultrasound-localized target within opaque fluid. In three experiments, the SF showed higher accuracy and lower variability in aiming and endpoint placements than did CUS. The SF, but not CUS, readily transferred to new targets and starting points for action. These effects were evident in visually guided action (needle and target continuously visible) and visually directed action (target alone visible). The results have application to learning to visualize surgical targets through ultrasound.

Our previous work has shown that visual distortion of subject-controlled goals can be used to manipulate force production and movement distance. Thus, visual distortion may be a helpful addition to robotic rehabilitation for brain-injured and stroke patients. In this work, we investigated the effects of distortion on unimpaired subjects learning a two-finger coordinated pinch task. Rather than distorting the subject's goal, we enhanced the apparent error attributable to one or both fingers. Distortion affected the distribution of attention between the two fingers; when the error enhancement was given to one finger, the performance of that finger improved relative to the other. This result indicates that enhanced visual error could be used to manipulate the performance and the coordination pattern between fingers to achieve desired movements during rehabilitation

We create a robotic rehabilitation environment that distorts the visual feedback of a movement representation to restore lost movements. The use of visual-feedback distortion produces a perceptual gap between the perceived (visual) and actual somatosensory experiences, in which movements can be manipulated without patients' knowledge. Previously, we reported the smallest amount of distortion that is imperceptible by measuring just noticeable difference (JND), the distortion size that is tolerable without detection, and the invariance of perceived physical effort under visual-feedback distortion. In this paper, we report the performance changes under visual-feedback distortion in a robotic rehabilitation environment. By interleaving trials with no visual feedback, we showed that the internal somatosensory representation of the movement goal changed along with visual distortion for unimpaired and motor-impaired subjects. In addition, a gradual change of visual-feedback distortion within one experiment allowed movement changes significantly larger than the JND. These changes were robust under sudden lack of visual feedback and even when subjects were informed of possible distortion ahead of time. Finally, preliminary results of a six-week rehabilitation paradigm are reported. Improvements in the patient's hand condition were found, particularly in the range of motion of the index and middle fingers

In previous work, we demonstrated that people reliably perceive variations in surface roughness when textured surfaces are explored with a rigid link between the surface and the skin [e.g., Klatzky and Lederman 1999; Klatzky et al. 2003]. Parallel experiments here investigated the potential of a force-feedback mouse to render surfaces varying in roughness. The stimuli were surfaces with alternating regions of high and low resistance to movement in the x (frontal) dimension (called ridges and grooves, respectively). Experiment 1 showed that magnitude ratings of roughness varied systematically with the spatial period of the resistance variation. Experiments 2 and 3 used a factorial design to disentangle the contributions of ridge and groove width. The stimuli constituted eight values of groove width at each of five levels of ridge width (Experiment 2) or the reverse (Experiment 3). Roughness magnitude increased with ridge width while remaining essentially invariant over groove width. Kinematic variations in exploration were observed across the surfaces. The data point to the promise of using inexpensive devices to create virtual textural variations under conditions of unconstrained exploration.

Klatzky and Lederman [2006] have shown that tangential resistive forces may be used to convey roughness in virtual textures using the WingMan force-feedback mouse. Modeling our experiment after this study, we directly examined the effect of viscous resistance on the perceived roughness magnitude of virtual gratings using a PHANTOM. For each virtual grating, the resistance level encountered at the ridges was varied by altering the viscosity coefficient. Perceived roughness systematically increased as the value of the viscosity coefficient was increased. The ridge-to-groove ratio contributed a small additional effect of microgeometry. These results suggest that simple models of viscous resistance may be used to simulate varying levels of surface roughness.

We report on two route guidance tasks using a highly accurate GPS receiver. Eight participants who were visually impaired or blind traveled two routes, one on a city sidewalk, and one in a city park. We tested and compared two types of spatial output devices that give route guidance information. One output display used a hand-held pointer, using a standard Talking Signs receiver that integrated the GPS signal information with the Talking Signs® signal information. This device gave travel instructions and on-course confirmation when pointed in the proper direction. The other spatial display used auditory virtual reality that presented the audible spatial information (waypoint direction and distance) through small air-tubes inserted into the ear. Travel times, distance, and errors were recorded. In addition, we tested users' ability to find precise locations, such as the intersections of small paths and a bus stop pole. Various subjective ratings were collected about blind participants' needs and perception of the various display and output options that they used. All subjects completed the tasks with both output displays, found all the waypoints and locations, and rated the two displays highly. The virtual sound display produced superior times overall and received slightly higher favorable ratings.

We present a novel psychophysical method for evaluating ultrasonography based on Real-Time Tomographic Reflection (RTTR), in comparison to Conventional Ultrasound (CUS). The method measures the user's perception of the location of an ultrasound-imaged target independently from assessing the action employed to reach it. Three experiments were conducted with the Sonic Flashlight (SF), an RTTR device, and CUS. The first two experiments determined subjects' perception of target location with a triangulation-by-pointing task. Depth perception with the SF was comparable to direct vision, while CUS caused considerable underestimation of target depth. Binocular depth information in the SF was shown to significantly contribute to its superiority. The third experiment tested subjects in an ultrasound-guided needle insertion task. Because the SF provides visualization of the target at its actual location, subjects performed insertions faster and more accurately by using the SF rather than CUS. Furthermore, the trajectory analysis showed that insertions with the SF generally went directly to the target along the desired path, while CUS often led to a large deviation from the correct path consistent with the observed underestimation of target depth. These findings lend great promise to the use of RTTR-based imaging in clinical practice and provide precise means of assessing efficacy.

In order to design a robotic rehabilitation environment using visual feedback distortion, we investigated in this study the limits and effects of visual feedback distortion for the elderly and the motor-impaired. To determine the minimum imperceptible amount of visual distortion, we measured the Just Noticeable Differences (JNDs) for force and position for elderly, unimpaired subjects; values of 31.0% (0.619 N) and 16.1% (5.01 mm), respectively, were obtained. JNDs of 46.0% (0.920 N) and 45.0% (14.8 mm) were measured for a motor-impaired individual. These JNDs were larger than corresponding measurements previously taken with young subjects, showing a decrease in discrimination ability with age and impairment. Visual distortion based on these values caused elderly subjects and the motor-impaired individual to increase their force production levels by 72.5% and 97.7%, respectively. These results were similar to those obtained with young subjects, but differences were observed on interspersed trials with no visual feedback. Poor discrimination abilities in elderly and impaired subjects and visual dominance in our environment for this subject group support our hypothesis that visual distortion can be an effective tool for rehabilitation in a robotic environment.

We report an experimental high-fidelity system for making psychophysical measurements on human operators performing real, virtual and real-remote 3D haptic manipulation tasks. Operators interact with task environments through six-degree-of-freedom (6-DOF) Lorentz magnetic levitation haptic devices. This arrangement allows the operator to exert and experience real, virtual, and real-remote forces/torques using the same 6-DOF master device. In the virtual task scenario, interactions are rendered haptically. In the real task scenario, the manipulation of the haptic device interacts by direct mechanics with a real environment. In the remote-real scenario, interactions with a remote real task environment are mediated through a 6-DOF Lorentz magnetic levitation slave device carried by a 6-DOF robot arm. In all three scenarios, visual feedback is provided by a graphical display. The system records accurate positions/orientations and forces/torques as a function of time. These records can be parsed automatically and analyzed off-line to evaluate operator performance.

We describe an experimental arrangement for comparison of user performance during a real and a virtual 3D peg-in-hole task. Tasks are performed using a unique six-degree-of-freedom (6-DOF) magnetic levitation haptic device. The arrangement allows a user to exert and experience real and virtual forces using the same 6-DOF device. During the virtual task, a peg and hole are rendered haptically, and visual feedback is provided through a graphical display. During the real task, a physical peg is attached to the underside of the haptic device. Using only real forces/torques, the peg is inserted into a hole in a plate attached to a force/torque sensor, while positions/orientations are measured by the haptic device. positions/orientations and forces/torques are recorded for both modes. Preliminary results indicate increased task time, larger total forces and more failures occur with the virtual task. Recorded data reveal user strategies that are similar for both tasks. Quantitative analysis of the strategies employed should lead to identification of significant factors in haptic interface design and haptic rendering techniques.

This paper describes the process of building a GIS for use in real time by blind travellers. Initially the components of a Personal Guidance System (PGS) for blind pedestrians are outlined. The location finding and database components of the system are then elaborated. Next follows a discussion of the environmental features likely to be used by blind travellers, and a discussion of the different path following and environmental learning modes that can be activated in the system. Developments such as personalizing the system and accounting for veering are also presented. Finally, possible competing schemes and problems related to the GIS component are examined.

We present a model relating materials to the sounds they make when struck. The model exploits a shape-invariant property of materials: the angle of internal friction. We demonstrate the utility of this model in both analyzing and synthesizing sounds generated by impulsive excitation. For the analysis of impact sounds, we present an active approach for discriminating different materials by impulsively contacting (hitting) them, and sensing the interpreting the resulting sound. Experimental results suggest that shape-invariance may be encoded in the functional form of the relation between and angle of internal friction and frequency. For the recreation of impact sounds, we identify two key problems-the impact sound reconstruction problem, and the impact sound synthesis problem-and discuss preliminary considerations in developing solutions.

We outline the design for a navigation system for the visually impaired and describe the progress we have made toward such a system. Our long-term goal is for a portable, self-contained system that will allow visually impaired individuals to travel through familiar and unfamiliar environments without the assistance of guides. The system, as it exists now, consists of the following functional components: (1) a means of determining the traveler's position and orientation in space, (2) a Geographic Information System comprising a detailed database of the surrounding environment and functions for automatic route planning and for selecting the database information desired by the user, and (3) the user interface.

This paper reports on progress towards the development of a personal guidance system for a blind navigator. The system is being developed in light of the unique problems facing the blind traveller, which are discussed first. The system consists of four modules: a locator unit, a detailed spatial database, an algorithm for path selection and a user interface. Suggestions are offered for alternative ways of handling the locator problem using a global positioning system or a database query system. The nature of the geographic information system (GIS) to be used as a host for the database is discussed, followed by brief statement of the criteria for choosing an algorithm for path selection and suggestions on types of user interfaces best suited to the system. A report on a pilot-study GIS is presented. Special problems of database and system design are highlighted.

This article describes a three-part empirical approach to understanding the development of expertise within the UNIX' operating system. We studied UNIX users with varying levels of expertise. The first part of our research attempted to ascertain the nature of their conceptualizations of the UNIX system. The second part measured users' performance in tasks requiring them to comprehend and produce UNIX commands. The third part was a longitudinal rather than cross-sectional analysis of the emergence of expertise. The conceptualization data suggest important differences in the models of UNIX structure formed by each group. Experts best represent the higher levels of the UNIX system; novices more fully represent the lower, more concrete levels of the system, including specific commands. UNIX users also differ markedly in performance, according to their history of use with the operating system. Only experts could successfully produce composite commands that required use of the distinctive features of UNIX (e.g., pipes and other redirection symbols), even though the intermediates and novices evidenced the component knowledge required for composite commands. This finding is somewhat surprising, inasmuch as these are fundamental design features of UNIX, and these features are taught in elementary classes. These data suggest, however, that these features can be used reliably only after extensive experience. The longitudinal data suggest that most subjects increased in expertise. However, expertise can decrease as a function of time, depending on system use. Those subjects who increased in expertise acquired the ability to produce the simple commands and represent the basic modules before they acquired knowledge of complex commands and advanced utilities. The nature of expertise is considered with respect to both system design and user characteristics, including users' conceptual models of system structure.