Systems’ latency – the time between user input and system response – slows down the human-computer interaction loop. Several studies revealed negative objective and subjective effects of high latency, typically treating latency as a constant delay. Because latency varies significantly in practice, recent work also assessed the effects of large and sudden latency changes. In practice, however, latency variations are small but frequent. As the effects of such variations are unclear, we investigate how small latency variations (± 50 ms) affect users’ performance and perceived task load for 2D target selection tasks with static and moving targets. For static targets, we found that latency variation causes significantly higher completion times and less efficient trajectories, however with small effect sizes. In contrast, we found no significant effects on any performance measure for moving targets. Our findings indicate that the effect of latency variation is generally very small and quickly disappears for non-trivial tasks.

Tables are focus points for social interactions and support everyday activities, such as learning, crafting, or dining. These physical interactions on and around the table may be augmented with digital information and tools projected onto the tabletop. For interaction with such projected information, touch input suffers from technical and interactional limitations. Pen input is a more robust alternative that does not suffer from Midas-touch problems. We developed a system for tracking the position of an IR-emitting pen tip on a planar surface with sub-millimeter resolution and an end-to-end latency of less than 30 ms. Distinguishing between drawing and hovering states is done by combining a stereoscopic camera setup and a machine-learning classifier. We demonstrate practical performance, uses and limitations through multiple studies and examples.

The latency of a text editor describes how long it takes from pressing a key to the corresponding letter appearing on the screen. It is well known that high latency affects how quickly authors can write and edit texts. In order to quantify the effects of latency on users’ performance and task load, we conducted a study in which 31 participants had to re-type or correct provided texts with a physical keyboard. Each participant completed each task once with a low latency of 20 ms and once with a high latency of 200 ms. We found that latency had no significant effect on users’ performance during the copy task, but correcting texts was affected significantly by high latency. Additionally, our results suggest that fast typers are more likely to notice latency than slow typers. Our findings regarding the effects of text input latency on users’ performance contributes to the existing body of research on latency in interactive systems.

In interactive systems high latency affects user performance and experience. This is especially problematic in video games. A large number of studies on this topic investigated the effects of constant, high latency. However, in practice, latency is never constant but varies by up to 100 ms due to variations in processing time and delays added by polling between system components. In a large majority of studies, these variations in latency are neither controlled for nor reported. Thus, it is unclear to which degree small, continuous variations in latency affect user performance. If these unreported variations had a significant impact, this might cast into doubt the findings of some studies. To investigate how latency variation affects player performance and experience in games, we conducted an experiment with 28 participants playing a first-person shooter. Participants played with two levels of base latency (50 ms vs. 150 ms) and variation (0 ms vs. 50 ms). As expected, high base latency significantly reduces player performance and experience. However, we found strong evidence that small variations in latency in the order of 50 ms, do not affect player performance significantly. Thus, our findings mitigate concerns that previous latency studies might have systematically ignored a confounding effect.

Real-time online interaction is the norm today. Tabletops and other dedicated interactive surface devices with direct input and tangible interaction can enhance remote collaboration, and open up new interaction scenarios based on mixed physical/virtual components. However, they are only available to a small subset of users, as they usually require identical bespoke hardware for every participant, are complex to setup, and need custom scenario-specific applications. We present SurfaceCast, a software toolkit designed to merge multiple distributed, heterogeneous end-user devices into a single, shared mixed-reality surface. Supported devices include regular desktop and laptop computers, tablets, and mixed-reality headsets, as well as projector-camera setups and dedicated interactive tabletop systems. This device-agnostic approach provides a fundamental building block for exploration of a far wider range of usage scenarios than previously feasible, including future clients using our provided API. In this paper, we discuss the software architecture of SurfaceCast, present a formative user study and a quantitative performance analysis of our framework, and introduce five example application scenarios which we enhance through the multi-user and multi-device features of the framework. Our results show that the hardware- and content-agnostic architecture of SurfaceCast can run on a wide variety of devices with sufficient performance and fidelity for real-time interaction.

Tangibles are small, graspable objects that act as input devices or physical representations of digital data. Oftentimes, it is desirable to track the position of tangibles on a surface and their relation to each other. However, outside-in tracking techniques - such as capacitive touchscreens or cameras - require setting up elaborate infrastructure and are prone to occlusion or interference. We propose Dothraki, an inside-out tracking technique for tangibles on flat surfaces. An optical mouse sensor embedded in the tangible captures a small (36×36 pixel / 1×1 mm), unique section of a black-and-white De-Bruijn dot pattern printed on the surface. Our system efficiently searches the pattern space in order to determine the precise location of the tangible with sub-millimeter accuracy. Our proof-of-concept implementation offers a recognition rate of up to 95%, robust error detection, an update rate of 14 Hz, and a low-latency relative tracking mode.

We propose a method for accurately and precisely measuring the intrinsic latency of input devices and document measurements for 36 keyboards, mice and gamepads connected via USB. Our research shows that devices differ not only in average latency, but also in the distribution of their latencies, and that forced polling at 1000 Hz decreases latency for some but not all devices. Existing practices - measuring end-to-end latency as a proxy of input latency and reporting only mean values and standard deviations - hide these characteristic latency distributions caused by device intrinsics and polling rates. A probabilistic model of input device latency demonstrates these issues and matches our measurements. Thus, our work offers guidance for researchers, engineers, and hobbyists who want to measure the latency of input devices or select devices with low latency.

For more than two decades, capacitive sensing has played a prominent role in human-computer interaction research. Capacitive sensing has become ubiquitous on mobile, wearable, and stationary devices - enabling fundamentally new interaction techniques on, above, and around them. The research community has also enabled human position estimation and whole-body gestural interaction in instrumented environments. However, the broad field of capacitive sensing research has become fragmented by different approaches and terminology used across the various domains. This paper strives to unify the field by advocating consistent terminology and proposing a new taxonomy to classify capacitive sensing approaches. Our extensive survey provides an analysis and review of past research and identifies challenges for future work. We aim to create a common understanding within the field of human-computer interaction, for researchers and practitioners alike, and to stimulate and facilitate future research in capacitive sensing.

Smart objects within instrumented environments offer an always available and intuitive way of interacting with a system. Connecting these objects to other objects in range or even to smartphones and computers, enables substantially innovative interaction and sensing approaches. In this paper, we investigate the concept of Capacitive Near-Field Communication to enable ubiquitous interaction with everyday objects in a short-range spatial context. Our central contribution is a generic framework describing and evaluating this communication method in Ubiquitous Computing. We prove the relevance of our approach by an open-source implementation of a low-cost object tag and a transceiver offering a high-quality communication link at typical distances up to 15 cm. Moreover, we present three case studies considering tangible interaction for the visually impaired, natural interaction with everyday objects, and sleeping behavior analysis.

Capacitive sensing allows the creation of unobtrusive user interfaces that are based on measuring the proximity to objects or recognizing their dielectric properties. Combining the data of many sensors, applications such as in-the-air gesture recognition, location tracking or fluid-level sensing can be realized. We present OpenCapSense, a highly flexible open-source toolkit that enables researchers to implement new types of pervasive user interfaces with low effort. The toolkit offers a high temporal resolution with sensor update rates up to 1 kHz. The typical spatial resolution varies between one millimeter at close object proximity and around one centimeter at distances of 35cm or above.

We present Vertibles, a set of Tangible User Interface (TUI) objects employing a vacuum-based adhesion effect. This effect allows attaching them to arbitrarily inclined surfaces, bringing the benefit of TUIs to vertical interactive surfaces. In contrast to other vertically attachable TUIs, Vertibles stick to a wide range of surface materials and work with optical as well as electric object tracking techniques for interactive surfaces. We present an overview of approaches for sticking objects onto vertical surfaces, describe the technical principle and properties of our solution, and document implementation details of a number of Vertibles prototypes.

The way we grasp an object depends on several factors, e.g. the intended goal or the hand's anatomy. Therefore, a grasp can convey meaningful information about its context. Inferring these factors from a grasp allows us to enhance interaction with grasp-sensitive objects. This paper highlights an grasp as an important source of meaningful context for human-computer interaction and gives an overview of prior work from other disciplines. This paper offers a basis and framework for further research and discussion by proposing a descriptive model of meaning in grasps. The GRASP model combines five factors that determine how an object is grasped: goal, relationship between user and object, anatomy, setting, and properties of the object. The model is validated both from an epistemological perspective and by applying it to scenarios from related work.

Time domain reflectometry, a technique originally used in diagnosing cable faults, can also locate where a cable is being touched. In this paper, we explore how to extend time domain reflectometry in order to touch-enable thin, modular, and deformable surfaces and devices. We demonstrate how to use this approach to make smart clothing and to rapid prototype touch-sensitive objects of arbitrary shape. To accomplish this, we extend time domain reflectometry in three ways: (1) Thin: We demonstrate how to run time domain reflectometry on a single wire. This allows us to touch-enable thin metal objects, such as guitar strings. (2) Modularity: We present a two-pin connector system that allows users to daisy chain touch-sensitive segments. We illustrate these enhancements with 13 prototypes and a series of performance measurements. (3) Deformability: We create deformable touch devices by mounting stretch-able wire patterns onto elastic tape and meshes. We present selected performance measurements.

This paper presents a method for prototyping grasp-sensitive surfaces using optical fibers. In this system one end of a fiber bundle is attached to an image sensor. The other ends of the individual fibers are attached to distinct points of a surface. Thus the image sensor can detect changes in light reception caused by a hand covering the surface. By emitting infrared light through the surface and measuring the amount of reflected light the system can also recognize touch and proximity. Mapping between pixels on the image sensor and fiber positions on the surface is generated by a relative calibration method. This setup allows to quickly build grasp-sensitive objects without electronics skills.

As mobile and tangible devices are getting smaller and smaller it is desirable to extend the interaction area to their whole surface area. The HandSense prototype employs capacitive sensors for detecting when it is touched or held against a body part. HandSense is also able to detect in which hand the device is held, and how. The general properties of our approach were confirmed by a user study. HandSense was able to correctly classify over 80 percent of all touches, discriminating six different ways of touching the device (hold left/right, pick up left/right, pick up at top/bottom). This information can be used to implement or enhance implicit and explicit interaction with mobile phones and other tangible user interfaces. For example, graphical user interfaces can be adjusted to the user's handedness.

In this paper we present a toolkit for realizing capacitive sensing applications for human-computer interaction in pervasive computing systems. We argue that capacitive sensors - due to their unique properties - are well suited for many pervasive and ubiquitous computing applications and scenarios. We describe the CapToolKit designed to rapidly realize prototypes and systems that are able to detect the presence of humans and objects. Our toolkit also allows the integration of 3D interaction with everyday objects as well as instrumented environments. We illustrate its capabilities by presenting several applications implemented using CapToolKit. The entire system will be open-sourced to allow utilization of our technology within other research projects. By building on the existing toolkit researchers are provided a foundation for developing their own sensor systems, algorithms, and applications