Well-designed haptic interactions can improve user experience, but implementing them is often challenging. Haptic authoring tools help alleviate this difficulty, but they are suited for use with different hardware, applications, and practitioners. As such, novices to haptic design need to determine which tools, if any, are suitable for their projects. Similarly, practitioners improving authoring tools and extending them to new contexts need to understand advancements made in prior work to identify opportunities for improvement. Unfortunately, the haptic authoring literature is disorganized, hindering practitioners from succeeding in these tasks. To remedy these shortcomings, we developed a taxonomy of haptic authoring tools, used it to identify gaps in the literature, and analyzed trends in the development of the tools. We also present the systematic literature review and study with thirteen haptics practitioners that were used to produce these results, then discuss what they mean for future research in haptic authoring.

Participants of one-to-many videoconferencing calls often experience a significant loss of affective feedback due to the limitations of the medium. To address this problem, we present Co-Here: a videoconferencing module that renders participant-driven animations that convey group affect without relying on categorical emotion detection or signaling. Co-Here consolidates and visualizes facial expressions and head movements of fellow videoconferencers, providing a sufficient medium for others to meaningfully construct emotional impressions. Our qualitative user study showed that the system helped users feel a sense of alignment with the emotions of others using the system. Co-Here had the most utility to presentation viewers, who reported that the system offered a low-attention alternative to gauging the sentiment of the crowd.

Our perception of emotion is highly contextual. Changes in the environment can affect our narrative framing, and thus augment our emotional perception of interlocutors. User environments are typically heavily suppressed due to the technical limitations of commercial videoconferencing platforms. As a result, there is often a lack of contextual awareness while participating in a video call, and this affects how we perceive the emotions of conversants. We present a videoconferencing module that visualizes the user's aural environment to enhance awareness between interlocutors. The system visualizes environmental sound based on its semantic and acoustic properties. We found that our visualization system was about 50% effective at eliciting emotional perceptions in users that was similar to the response elicited by environmental sound it replaced.The contributed system provides a unique approach to facilitate ambient awareness on an implicit emotional level in situations where multimodal environmental context is suppressed.

The need to generate convincing simulation of voices often arises in the context of avatar therapy, a treatment approach for disorders such as schizophrenia. This treatment involves patients interacting with simulations of the entity they imagine to be responsible for the voices they hear, for which there is often no external reference available. However, in such scenarios, there is little knowledge of how to design and reproduce these voices in a convincing manner. Existing voice manipulation interfaces are often complex to use, and highly limited in their ability to modify vocal characteristics beyond small adjustments. To address these challenges, we designed a framework that allows users to explore and select from a large set of voices, and thereafter manipulate the voice(s) to converge towards an effective match for one they have in mind. We demonstrated both the usability and superior performance of this system compared to existing voice manipulation interfaces.

With the recent popularization of generative frameworks for producing photorealistic face images, we now have the ability to create a convincing graphical match for any particular individual. It is unrealistic, however, to rely solely on such generative methods to randomly produce the facial characteristics we are seeking. Instead, manipulation of facial attributes in the latent space, enabled by the InterFaceGAN framework, allows us to “tweak” these characteristics in the desired direction to improve the quality of the match. The challenge in this process is that attribute entanglement leads to a change of one feature having an undesirable impact on others. We explore several strategies to improve the results of these manipulations, and demonstrate how the automatic conditioning of attributes can be used to minimize the impact of such entanglement, and further, allow for improved control over complex (non-binary) attributes such as race or face shape.

The application of haptic technology in entertainment systems, such as Virtual Reality and 4D cinema, enables novel experiences for users and drives the demand for efficient haptic authoring systems. Here, we propose an automatic multimodal vibrotactile content creation pipeline that substantially improves the overall hapto-audiovisual (HAV) experience based on contextual audio and visual content from movies. Our algorithm is implemented on a low-cost system with nine actuators attached to a viewing chair and extracts significant features from video files to generate corresponding haptic stimuli. We implemented this pipeline and used the resulting system in a user study (n = 16), quantifying user experience according to the sense of immersion, preference, harmony, and discomfort. The results indicate that the haptic patterns generated by our algorithm complement the movie content and provide an immersive and enjoyable HAV user experience. This further suggests that the pipeline can facilitate the efficient creation of 4D effects and could therefore be applied to improve the viewing experience in home theatre environments.

Today's smartphone notification systems are incapable of determining whether a notification has been successfully perceived without explicit interaction from the user. If the system incorrectly assumes that a notification has not been perceived, it may repeat it redundantly, disrupting the user and others (e.g., phone ringing). Or, if it incorrectly assumes that a notification was perceived, and therefore fails to repeat it, the notification will be missed altogether (e.g., text message). Results from a laboratory study confirm, for the first time, that both vibrotactile and auditory smartphone notifications induce skin conductance responses (SCR), that the induced responses differ from that of arbitrary stimuli, and that they could be employed to predict perception of smartphone notifications after their presentation using wearable sensors.

Experience Sampling Methods (ESM) allow the timely collection of subjective self-reports that would otherwise be impossible to measure accurately in ecologically valid scenarios. Recent work suggests that unlock journaling allowed the collection of more data points per day, was faster and perceived as being less intrusive by participants than notification-based ESM. This work extends the unlock journaling field by introducing a novel lockscreen data collection mechanism harnessing an increasingly popular authentication mechanism: the fingerprint sensor. Results collected during a twelve-day user study with fingerprint sensor users show that fingerprint sensor gesture reporting compares favorably to Slide-to-X approaches. The proposed gestural interface was subjectively perceived as being the fastest, least intrusive, and overall most preferred interface, in addition to offering the highest response compliance. By offering a reporting mechanism better aligned with modern smartphone unlocking habits, this work encourages the deployment of unlock journaling in the wild.

Machine learning (ML) has become an important topic for students across disciplines to understand because of its useful applications and its societal impacts. At the same time, there is little existing work on ML education, particularly about teaching ML to non-majors. This paper presents an exploration of the pedagogical content knowledge (PCK) for teaching ML to non-majors. Through ten interviews with instructors of ML courses for non-majors, we inquired about student preconceptions as well as what students find easy or difficult about learning ML. We identified PCK in the form of three preconceptions and five barriers faced by students, and six pedagogical tactics adopted by instructors. The preconceptions were found to concern themselves more with ML's reputation rather than its inner workings. Student barriers included underestimating human decision in ML and conflating human thinking with computer processing. Pedagogical tactics for teaching ML included strategically choosing datasets, walking through problems by hand, and customizing to the domain(s) of students. As we consider the lessons from these findings, we hope that this will serve as a first step toward improving the teaching of ML to non-majors.

Given its societal impacts and applications to numerous fields, machine learning (ML) is an important topic to understand for many students outside of computer science and statistics. However, machine-learning education research is nascent, and research on this subject for non-majors thus far has only focused on curricula and courseware. We interviewed 10 instructors of ML courses for non-majors, inquiring as to what their students find both easy and difficult about machine learning. While ML has a reputation for having algorithms that are difficult to understand, in practice our participating instructors reported that it was not the algorithms that were difficult to teach, but the higher-level design decisions. We found that the learning goals that participants described as hard to teach were consistent with higher levels of the Structure of Observed Learning Outcomes (SOLO) taxonomy, such as making design decisions and comparing/contrasting models. We also found that the learning goals that were described as easy to teach, such as following the steps of particular algorithms, were consistent with the lower levels of the SOLO taxonomy. Realizing that higher-SOLO learning goals are more difficult to teach is useful for informing course design, public outreach, and the design of educational tools for teaching ML.

Operating power tools over extended periods of time can pose significant risks to humans, due to the strong forces and vibrations they impart to the limbs. Telemanipulation systems can be employed to minimize these risks, but may impede effective task performance due to the reduced sensory cues they typically convey. To address this shortcoming, we explore the benefits of augmenting these cues with the addition of audition, vibration, and force feedback, and evaluate them on users' performance in a VR mechanical assembly task employing a simulated impact wrench. Our research focuses on the utility of vibrotactile feedback, rendered as a simplified and attenuated version of the vibrations experienced while operating an actual impact wrench. We investigate whether such feedback can serve to enhance the operator's awareness of the state of the tool, as well as a proxy for the forces experienced during collisions and coupling, while operating the tool an actual impact wrench. Results from our user study comparing feedback modalities confirm that the introduction of vibrotactile, in addition to auditory feedback can significantly improve user performance as assessed by completion time. However, the addition of force feedback to these two modalities did not further improve performance.

Despite considerable prior work exploring foot-based interaction techniques, direct comparisons of the performance of these approaches have been lacking. Here, we compare the performance of the two most common approaches found in previous studies: rocking (applying pressure to different parts of the foot) versus rotating and sliding, considering the use case of a hands-free interface intended for seated musicians. Participants performed a number of representative operations, such as setting the tempo of a metronome, using the two strategies. Results indicate superiority of the rotating and sliding approach, both in completion time and responses to NASA TLX questionnaires, although rocking was preferred by some participants due to its ergonomics and subtle movements required for parameter-controlling tasks. Beyond the comparison itself, the decisions we faced related to menu design and feedback for our use case may offer helpful insight for the design of future foot-based interfaces.

While several foot-controlled pointing devices have been explored as alternatives to conventional interfaces, we are interested in whether such devices can achieve higher performance with the addition of variable friction. Users wore our variable-friction prototype shoe on their right foot, which they slid on a low-friction surface to control a mouse cursor. Two interface modes were evaluated: constant (CF) and variable friction (VF), under the ISO 9241-9 standard for pointing device evaluation. For the variable-friction modality, target regions were high friction to provide sliding resistance cues. Our findings confirmed that variable-friction foot-controlled pointing can achieve throughput competitive with a range of hand-controlled devices. This suggests the potential for taking advantage of foot input for simple pointing tasks, in particular when the hands are overloaded. With respect to other foot-controlled pointing systems, our implementation offered improved performance and comparable error rates. In addition, the analysis provided further insight into the design of foot-controlled input devices.

Considerable effort has been invested in the development of effective emotion and behavior recognition techniques. In comparison, little work has been devoted to technologies that can be used to induce specific emotional and behavioral responses, with most such research relying on the presentation of video or images. In this article, we propose a novel technique for the elicitation of emotion based on audio-tactile stimulation. Taking advantage of the relationship between tickling, laughter and emotional states, we conducted an experiment to map the perception of the tickle sensation as a function of vibrotactile stimulation frequency, quantify the effect of hearing laughter stimulus on the perceived intensity of the tactile experience, and assess the potential of the proposed multimodal approach to induce observable mirthful responses. Experimental evidence shows that the perceived intensity of the auditory laughter stimulus has a repeatable scaling effect on the tickle sensation and that the proposed audio-tactile stimulation is a promising approach to laughter elicitation. These findings may inform the design of future multimodal affective interfaces by allowing a more informed prediction of induced emotional and behavioral responses.

Many variables have been shown to impact whether a vibration stimulus will be perceived. We present a user study that takes into account not only previously investigated predictors such as vibration intensity and duration along with the age of the person receiving the stimulus, but also the amount of motion, as measured by an accelerometer, at the site of vibration immediately preceding the stimulus. This is a more specific measure than in previous studies showing an effect on perception due to gross conditions such as walking. We show that a logistic regression model including prior acceleration is significantly better at predicting vibration perception than a model including only vibration intensity, duration and participant age. In addition to the overall regression, we discuss individual participant differences and measures of classification performance for real-world applications. Our expectation is that haptic interface designers will be able to use such results to design better vibrations that are perceivable under the user's current activity conditions, without being annoyingly loud or jarring, eventually approaching ``perceptually equivalent' feedback independent of motion.

Although multiple GPS-based navigation applications exist for the visually impaired, these are typically poorly suited for in-situ exploration, require cumbersome hardware, lack support for widely accessible geographic databases, or do not take advantage of advanced functionality such as spatialized audio rendering. These shortcomings led to our development of a novel spatial awareness application that leverages the capabilities of a smartphone coupled with worldwide geographic databases and spatialized audio rendering to convey surrounding points of interest. This paper describes the usability evaluation of our system through a task-based study and a longer-term deployment, each conducted with six blind users in real settings. The findings highlight the importance of testing in ecologically valid contexts over sufficient periods to face real-world challenges, including balancing quality versus quantity for audio information, overcoming limitations imposed by sensor accuracy and quality of database information, and paying appropriate design attention to physical interaction with the device.

Falls represent a major challenge to mobility for the elderly community, a point that has motivated various studies of balance failures. To support this work, we are interested in mechanisms for the synthesis of ground environments that can be controlled to exhibit dynamic friction characteristics. As a first step, we investigate the design and development of such a variable-friction device, a hybrid locomotion interface using a cable-driven vibrotactile mechanism. Measurements on our prototype, consisting of an aluminum tile covered with low-friction polytetrafluoroethylene (PTFE), demonstrate that it can effectively simulate a low coefficient of static friction. As part of the design, we also investigated the role that induced vibration plays in modifying the coefficient of friction. Measurements of sliding on a PTFE-covered tile in a tilted configuration showed a significant influence of normal low-frequency vibration, particularly for frequencies around 20 Hz, regardless of the user's weight.

In this paper, we present TeleHuman, a cylindrical 3D display portal for life-size human telepresence. The TeleHuman 3D videoconferencing system supports 360 degree motion parallax as the viewer moves around the cylinder and optionally, stereoscopic 3D display of the remote person. We evaluated the effect of perspective cues on the conveyance of nonverbal cues in two experiments using a one-way telecommunication version of the system. The first experiment focused on how well the system preserves gaze and hand pointing cues. The second experiment evaluated how well the system conveys 3D body postural information. We compared 3 perspective conditions: a conventional 2D view, a 2D view with 360 degree motion parallax, and a stereoscopic view with 360 degree motion parallax. Results suggest the combined presence of motion parallax and stereoscopic cues significantly improved the accuracy with which participants were able to assess gaze and hand pointing cues, and to instruct others on 3D body poses. The inclusion of motion parallax and stereoscopic cues also led to significant increases in the sense of social presence and telepresence reported by participants.

We propose a simple preconditioning method for accelerating the solution of edge-preserving image super-resolution (SR) problems in which a linear shift-invariant point spread function is employed. Our technique involves reordering the high-resolution (HR) pixels in a similar manner to what is done in preconditioning methods for quadratic SR formulations. However, due to the edge preserving requirements, the Hessian matrix of the cost function varies during the minimization process. We develop an efficient update scheme for the preconditioner in order to cope with this situation. Unlike some other acceleration strategies that round the displacement values between the low-resolution (LR) images on the HR grid, the proposed method does not sacrifice the optimality of the observation model. In addition, we describe a technique for preconditioning SR problems involving rational magnification factors. The use of such factors is motivated in part by the fact that, under certain circumstances, optimal SR zooms are nonintegers. We show that, by reordering the pixels of the LR images, the structure of the problem to solve is modified in such a way that preconditioners based on circulant operators can be used.

Mosaicing is largely dependent on the quality of registration among the constituent input images. Parallax and object motion present challenges to image registration, leading to artifacts in the result. To reduce the impact of these artifacts, traditional image mosaicing approaches often impose planar scene constraints or rely on purely rotational camera motion or dense sampling. However, these requirements are often impractical or fail to address the needs of all applications. Instead, taking advantage of depth cues and a smooth transition criterion, we achieve significantly improved mosaicing results for static scenes, coping effectively with nontrivial parallax in the input. We extend this approach to the synthesis of dynamic video mosaics, incorporating foreground/background segmentation and a consistent motion perception criterion. Although further additions are required to cope with unconstrained object motion, our algorithm can synthesize a perceptually convincing output, conveying the same appearance of motion as seen in the input sequences.

We present techniques to enable users to interact on foot with simulated natural ground surfaces, such as soil or ice, in immersive virtual environments. Position and force estimates from in-floor force sensors are used to synthesize plausible auditory and vibrotactile feedback in response. Relevant rendering techniques are discussed in the context of walking on a virtual frozen pond.

Image restoration is an ill-posed problem that must be regularized in order to reduce noise amplification in the restored image. Although quadratic penalty terms allow for fast restoration algorithms based on the fast Fourier transform (FFT), they often lead to images whose discontinuities are not well preserved. On the other hand, edge-preserving penalty terms can produce better results at the expense of computational efficiency. A restoration technique exploiting the Woodbury matrix identity was recently presented. However, its performance decreases when the number of discontinuities becomes significant. To overcome this problem, we propose a simple preconditioner to be employed in conjunction with the preconditioned nonlinear conjugate gradient method. Experiments are employed to demonstrate the effectiveness of our approach.

Traditional image-based mosaicing deals with the problem of parallax by imposing constraints of a parallax-free camera configuration or requiring a dense sampling of the scene. These solutions are often impractical or fail to address the needs of the application. Instead, taking advantage of depth cues and a criterion of smooth transitions, we achieve significantly improved mosaicing results for static scenes, coping effectively with non-trivial parallax in the input. Furthermore, by incorporating a criterion of consistent motion perception, we demonstrate progress on mosaicing of dynamic scenes without introducing artifacts. Although further additions are required to cope with unconstrained object motion, our algorithm can synthesize perceptually convincing dynamic mosaics, conveying the same appearance of object motion as seen in the original sequences.

Image mosaicing constructs a wide field-of-view result from multiple source frames. In order to ensure a perceptually correct result, mosaicing typically requires either a planar or near-planar scene, parallax-free camera motion between source frames, or a dense sampling of the scene. When these conditions are not satisfied, various artifacts may result. A novel mosaicing approach is introduced that overcomes these limitations, building on the techniques of image-based rendering and manifold mosaicing, while permitting the synthesis of an effective mosaic of a non-planar scene from a sparse set of translated cameras. Our method first generates a series of intermediate virtual frames to reduce the disparities between neighboring images. Next, a series of vertical slices are chosen from the array of both real and virtual frames and connected according to a cost function that maximizes the similarity between adjacent slices. Experimental results indicate significant improvements over competing methods. 1

When an occluding object, such as a person, stands between a projector and a display surface, a shadow results. We can compensate by positioning multiple projectors so they produce identical and overlapping images and by using a system to locate shadows. Existing systems work by detecting either the shadows or the occluders. Shadow detection methods cannot remove shadows before they appear and are sensitive to video projection, while current occluder detection methods require near infrared cameras and illumination. Instead, we propose using a camera-based object tracker to locate the occluder and an algorithm to model the shadows. The algorithm can adapt to other tracking technologies as well. Despite imprecision in the calibration and tracking process, we found that our system performs effective shadow removal with sufficiently low processing delay for interactive applications with video projection.

Super-resolution (SR) is a technique that generates a high-resolution (HR) image from a set of low-resolution (LR) ones. Previous preconditioning methods for SR did not consider the case of rational magnification factors. In this paper, a method for preconditioning SR problems involving such factors is presented. We show that by reordering the pixels of the observed LR images, the structure of the linear problem to solve is modified in such a way that preconditioners based on circulant operators can be used. Simulations with magnification factors of practical interest demonstrate the effectiveness of our approach.

The computational understanding of continuous human movement plays a significant role in diverse emergent applications in areas ranging from human computer interaction to physical and neuro-rehabilitation. Non-visual feedback can aid the continuous motion control tasks that such applications frequently entail. An architecture is introduced for enabling interaction with a system that furnishes a number of gestural affordances with assistive feedback. The approach combines machine learning techniques for understanding a user's gestures with a method for the display of salient features of the underlying inference process in real time. Methods used include a particle filter to track multiple hypotheses about a user's input as the latter is unfolding, together with models of the nonlinear dynamics intrinsic to the movements of interest. Non-visual feedback in this system is based on a presentation of error features derived from an estimate of the sampled time varying probability that the user's gesture corresponds to the various tracked state trajectories in the different dynamical systems. We describe applications to interactive systems for human gait analysis and rehabilitation, a domain of considerable current interest in the movement sciences and health care.

The diffusion framework exhibits many promising properties for the pur-poses of junction analysis. In its most common form, images are diffused either isotropically or with respect to gradient information in an anisotropical fashion. This information is then collected into an orientational distribution function (ODF) and the resulting features are modeled as ’X’, ’Y ’ or ’T’-shaped junctions. Specific to the spatio-temporal domain, and in particular, a 2D spatio-temporal slice, points of kinetic-based occlusion are identified by T-junctions while points of kinetic-transparency form X-junctions. The challenge is that most forms of diffusion are symmetric in their representa-tion and are unable to properly distinguish between these two junction types. This work proposes to diffuse information asymmetrically and investigates the differences between weighting the iterative diffusion isotropically versus as an ODF-shaped region of influence function. 1

Temporal superresolution produces a video sequence of high temporal resolution (HTR), i.e. frame rate, from a set of video sequences of low temporal resolution (LTR). A method for preconditioning large systems of linear equations arising in temporal video superresolution problems is presented. We show that circulant block preconditioners can accelerate the convergence of iterative techniques such as the conjugate gradient method when solving such problems. Simulations demonstrate the effectiveness of our approach.

Immersive Environments offer users the experience of being submerged in a virtual space, effectively transcending the boundary between the real and virtual world. We present a framework for visualization and manipulation of 3D virtual environments in which users need not resort to the awkward command vocabulary of traditional keyboard-and-mouse interaction. We have adapted the transparent toolglass paradigm as a gestural interface widget for a spatially immersive environment. To serve that purpose, we have implemented a bimanual gesture interpreter to recognize and translate a user's actions into commands for control of these widgets. In order to satisfy a primary design goal of keeping the user completely untethered, we use purely video-based tracking techniques.

Detecting human arm motion in a typical classroom environment is a challenging task due to the noisy and highly dynamic background, varying light conditions, as well as the small size and multiple number of possible matched objects. A robust vision system that can detect events of students' hands being raised for asking questions is described. This system is intended to support the collaborative demands of distributed classroom lecturing and further serve as a test case for real-time gesture recognition vision systems. Various techniques including temporal and spatial segmentation, skin color identification, as well as shape and feature analysis are investigated and discussed. Limitations and problems are also analyzed and testing results are illustrated.

Article Free Access Share on Beating the limitations of camera-monitor mediated telepresence with extra eyes Authors: Kimiya Yamaashi Hitachi Research Laboratory, Hitachi Ltd., 7-1-1 Omika-cho Hitachi-shi, Ibaraki-ken 319-12 Japan Hitachi Research Laboratory, Hitachi Ltd., 7-1-1 Omika-cho Hitachi-shi, Ibaraki-ken 319-12 JapanView Profile , Jeremy R. Cooperstock Telepresence Project, University of Toronto, 10 King's College Road, Toronto ON M5C 3G4, Canada Telepresence Project, University of Toronto, 10 King's College Road, Toronto ON M5C 3G4, CanadaView Profile , Tracy Narine Telepresence Project, University of Toronto, 10 King's College Road, Toronto ON M5C 3G4, Canada Telepresence Project, University of Toronto, 10 King's College Road, Toronto ON M5C 3G4, CanadaView Profile , William Buxton Telepresence Project, University of Toronto, 10 King's College Road, Toronto ON M5C 3G4, Canada and Alias/Wavefront, 110 Richmond Street East, Toronto ON M5C 1P1, Canada Telepresence Project, University of Toronto, 10 King's College Road, Toronto ON M5C 3G4, Canada and Alias/Wavefront, 110 Richmond Street East, Toronto ON M5C 1P1, CanadaView Profile Authors Info & Claims CHI '96: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsApril 1996 Pages 50–57https://doi.org/10.1145/238386.238402Published:13 April 1996Publication History 26citation738DownloadsMetricsTotal Citations26Total Downloads738Last 12 Months38Last 6 weeks9 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteView all FormatsPDF

A basic tenet of "Ubiquitous computing" (Weiser, 1993 [13]) is that technology should be distributed in the environment (ubiquitous), yet invisible, or transparent.In practice, resolving the seeming paradox arising from the joint demands of ubiquity and transparency is less than simple.This paper documents a case study of attempting to do just that.We describe our experience in developing a working conference room which is equipped to support a broad class of meetings and media.After laying the groundwork and establishing the context in the Introduction, we describe the evolution of the room.Throughout, we attempt to document the rationale and motivation.While derived from a limited domain, we believe that the issues that arise are of general importance, and have strong implications on future research.