Over the past decade, real-time embedded virtual presence systems have evolved from latency-constrained immersive prototypes into highly adaptive, ethically grounded hybrid infrastructures that bridge the physical and digital worlds. This article surveys this progression from an HCI perspective, structured across five timelines from 2015 to 2025. Timeline 1 charts the emergence of consumer-grade AR/VR systems, highlighting frameless rendering, distributed latency models, and avatar realism as critical enablers. Timeline 2 shifts toward context-aware systems that adapt interfaces and behavior to user state, cognitive load, and environmental cues. Timeline 3 introduces adaptive and AI-driven architectures, embedding learning and emotion recognition to personalize virtual interaction. Timeline 4 advances human–agent interaction through real-time multimodal synchronization, haptic fidelity, and cooperative dialogue prediction. Finally, Timeline 5 consolidates MR, IoT, Digital Twins, and edge computing into seamlessly integrated hybrid systems capable of bidirectional control and real-time synchronization. Across all timelines, challenges persist in synchronization, multimodal fusion, and ethical governance. This survey identifies technical trajectories, systemic transitions, and open research challenges that shape the future of real-time embedded virtual presence.

Videos are dynamic and multi-modal compared to other types of content, making automatic filtering difficult, which is why content moderators play a crucial role. However, video content moderators are exposed to more profound emotional labor because videos contain rich visual information, sometimes including even harmful content, such as violent or terrifying scenes. In this work, we explore the effect of six intervention techniques on alleviating negative emotions during video content moderation tasks. We conducted one online crowdsourcing experiment and two controlled user studies to find out that (i) interleaving with positive videos or (ii) cartoonization could significantly reduce negative emotions in the moderators. Participants reported that the advantages of these approaches are in helping reduce negative emotions at the time of moderation while existing approaches focus on post-task activities (e.g., relaxation, talking with others, or getting a hobby). We discuss the applicability of our findings to broader tasks, including improvement in intervention techniques.

Computer mice are widely used today as the primary input device in competitive video games. If a player exhibits more wrist rotation than other players when moving the mouse laterally, the player is said to have stronger wrist-aiming habits. Despite strong public interest, there has been no affordable technique to quantify the extent of a player’s wrist-aiming habits and no scientific investigation into how the habits affect player performance and workload. We present a reliable and affordable technique to quantify the extent of a player’s wrist-aiming habits using a mouse equipped with two optical sensors (i.e., a dual-sensor mouse). In two user studies, we demonstrate the reliability of the technique and examine the relationship between wrist-aiming habits and player performance or workload. In summary, player expertise and mouse sensitivity significantly impacted wrist-aiming habits; the extent of wrist-aiming showed a positive correlation with upper limb workload.

A significant amount of research has recently been conducted on user performance in so-called temporal pointing tasks, in which a user is required to perform a button input at the timing required by the system. Consecutive temporal pointing (CTP), in which two consecutive button inputs must be performed while satisfying temporal constraints, is common in modern interactions, yet little is understood about user performance on the task. Through a user study involving 100 participants, we broadly explore user performance in a variety of CTP scenarios. The key finding is that CTP is a unique task that cannot be considered as two ordinary temporal pointing processes. Significant effects of button input method, motor limitations, and different hand use were also observed.

This study investigates the effect of Lift-off Distance (LoD) on a computer mouse, which refers to the height at which a mouse sensor stops tracking when lifted off the surface. Although a low LoD is generally preferred to avoid unintentional cursor movement in mouse lifting (=clutching), especially in first-person shooter games, it may reduce tracking stability. We conducted a psychophysical experiment to measure the perceptible differences between LoD levels and quantitatively measured the unintentional cursor movement error and tracking stability at four levels of LoD while users performed mouse lifting. The results showed a trade-off between movement error and tracking stability at varying levels of LoD. Our findings offer valuable information on optimal LoD settings, which could serve as a guide for choosing a proper mouse device for enthusiastic gamers.

In first-person shooters (FPS), professional players (a.k.a., Gosu) outperform amateur players. The secrets behind the performance of professional FPS players have been debated in online communities with many conjectures; however, attempts of scientific verification have been limited. We addressed this conundrum through a data-collection study of the gameplay of eight professional and eight amateur players in the commercial FPS Counter-Strike: Global Offensive. The collected data cover behavioral data from six sensors (motion capture, eye tracker, mouse, keyboard, electromyography armband, and pulse sensor) and in-game data (player data and event logs). We examined conjectures in four categories: aiming, character movement, physicality, and device and settings. Only 6 out of 13 conjectures were supported with statistically sufficient evidence.

Despite the prevalence of shape-writing (gesture typing, swype input, or swiping for short) as a text entry method, there are currently no public datasets available. We report a large-scale dataset that can support efforts in both empirical study of swiping as well as the development of better intelligent text entry techniques. The dataset was collected via a web-based custom virtual keyboard, involving 1,338 users who submitted 11,318 unique English words. We report aggregate-level indices on typing performance, user-related factors, as well as trajectory-level data, such as the gesture path drawn on top of the keyboard or the time lapsed between consecutively swiped keys. We find some well-known effects reported in previous studies, for example that speed and error are affected by age and language skill. We also find surprising relationships such that, on large screens, swipe trajectories are longer but people swipe faster.

In gaming, accurately rendering input signals on a display is crucial, both spatially and temporally. However, the asynchronicity between the input and output signal frequencies results in unstable responses called jitter. A recent research modeled this jitter mathematically [2]; however, the effect of jitter on human performance is unknown. In this study, we investigated the empirical effect of asynchronicity-induced jitter using a state-of-the-art high-performance mouse and monitor device. In the first part, perceptual user experience under different jitter levels was examined using the ISO 4120:2004 triangle test protocol, and a jitter of over 0.3 ms could be perceived by sensitive subjects. In the second part, we measured the pointing task performance for different jitter levels using the ISO 9241-9 (i.e., Fitts’ law) test, and found that the pointing performance was unaffected up to a jitter of 1 ms. Finally, we recommended display and mouse combinations based on our results, which indicated the need for a higher mouse polling rate than that of the current standard 1000-Hz USB mouse.

Designing a push-button with desired sensation and performance is challenging because the mechanical construction must have the right response characteristics. Physical simulation of a button's force-displacement (FD) response has been studied to facilitate prototyping; however, the simulations' scope and realism have been limited. In this paper, we extend FD modeling to include vibration (V) and velocity-dependence characteristics (V). The resulting FDVV models better capture tactility characteristics of buttons, including snap. They increase the range of simulated buttons and the perceived realism relative to FD models. The paper also demonstrates methods for obtaining these models, editing them, and simulating accordingly. This end-to-end approach enables the analysis, prototyping, and optimization of buttons, and supports exploring designs that would be hard to implement mechanically.

Computer mice have their displacement sensors in various locations (center, front, and rear). However, there has been little research into the effects of sensor position or on engineering approaches to exploit it. This paper first discusses the mechanisms via which sensor position affects mouse movement and reports the results from a study of a pointing task in which the sensor position was systematically varied. Placing the sensor in the center turned out to be the best compromise: improvements over front and rear were in the 11-14% range for throughput and 20--23% for path deviation. However, users varied in their personal optima. Accordingly, variable-sensor-position mice are then presented, with a demonstration that high accuracy can be achieved with two static optical sensors. A virtual sensor model is described that allows software-side repositioning of the sensor. Individual-specific calibration should yield an added 4% improvement in throughput over the default center position.

A well-designed control-to-display gain function can improve pointing performance with indirect pointing devices like trackpads. However, the design of gain functions is challenging and mostly based on trial and error. AutoGain is a novel method to individualize a gain function for indirect pointing devices in contexts where cursor trajectories can be tracked. It gradually improves pointing efficiency by using a novel submovement-level tracking+optimization technique that minimizes aiming error (undershooting/overshooting) for each submovement. We first show that AutoGain can produce, from scratch, gain functions with performance comparable to commercial designs, in less than a half-hour of active use. Second, we demonstrate AutoGain's applicability to emerging input devices (here, a Leap Motion controller) with no reference gain functions. Third, a one-month longitudinal study of normal computer use with AutoGain showed performance improvements from participants' default functions.

Text revision is an important task to ensure the accuracy of text content. Revising text on mobile devices is cumbersome and time-consuming due to the imprecise caret control and the repetitive use of the backspace. We present Swap, a novel replacement-based technique to facilitate text revision on mobile devices. We conducted two user studies to validate the feasibility and the effectiveness of Swap compared to traditional text revision techniques. Results showed that Swap reduced efforts in caret control and repetitive backspace pressing during the text revision process. Most participants preferred to use the replacement-based technique rather than backspace and caret. They also commented that the new technique is easy to learn, and it makes text revision rapid and intuitive.

Effects of unintended latency on gamer performance have been reported. End-to-end latency can be corrected by post-input manipulation of activation times, but this gives the player unnatural gameplay experience. For moving-target selection games such as Flappy Bird, the paper presents a predictive model of latency on error rate and a novel compensation method for the latency effects by adjusting the game's geometry design -- e.g., by modifying the size of the selection region. Without manipulation of the game clock, this can keep the user's error rate constant even if the end-to-end latency of the system changes. The approach extends the current model of moving-target selection with two additional assumptions about the effects of latency: (1) latency reduces players' cue-viewing time and (2) pushes the mean of the input distribution backward. The model and method proposed have been validated through precise experiments.

This paper presents a large-scale dataset on mobile text entry collected via a web-based transcription task performed by 37,370 volunteers. The average typing speed was 36.2 WPM with 2.3% uncorrected errors. The scale of the data enables powerful statistical analyses on the correlation between typing performance and various factors, such as demographics, finger usage, and use of intelligent text entry techniques. We report effects of age and finger usage on performance that correspond to previous studies. We also find evidence of relationships between performance and use of intelligent text entry techniques: auto-correct usage correlates positively with entry rates, whereas word prediction usage has a negative correlation. To aid further work on modeling, machine learning and design improvements in mobile text entry, we make the code and dataset openly available.

Interaction on the surface usually limited in mechanical movement and does not have surface texture, which results in a lack of haptic feedback. Various techniques have been introduced to supplement this limitation. The main objective of this tutorial is to let the ISS community experience those technologies in hand. By its nature, the haptic interface is hardly explainable with a document, video, and audio. We provide a unique opportunity for the ISS community to feel and experience the wide range of haptic techniques on the surface. In specific, the tutorial will cover the following topics. 1) haptic feedback on a small mobile device 2) Texture rendering on a larger surface 3) An illusion of compliance on a rigid device 4) Pin-array display for information transfer.

The activation point of a button is defined as the depth at which it invokes a make signal. Regular buttons are activated during the downward stroke, which occurs within the first 20 ms of a press. The remaining portion, which can be as long as 80 ms, has not been examined for button activation for reason of mechanical limitations. The paper presents a technique and empirical evidence for an activation technique called Impact Activation, where the button is activated at its maximal impact point. We argue that this technique is advantageous particularly in rapid, repetitive button pressing, which is common in gaming and music applications. We report on a study of rapid button pressing, wherein users' timing accuracy improved significantly with use of Impact Activation. The technique can be implemented for modern push-buttons and capacitive sensors that generate a continuous signal.

This paper investigates a common task requiring temporal precision: the selection of a rapidly moving target on display by invoking an input event when it is within some selection window. Previous work has explored the relationship between accuracy and precision in this task, but the role of visual cues available to users has remained unexplained. To expand modeling of timing performance to multimodal settings, common in gaming and music, our model builds on the principle of probabilistic cue integration. Maximum likelihood estimation (MLE) is used to model how different types of cues are integrated into a reliable estimate of the temporal task. The model deals with temporal structure (repetition, rhythm) and the perceivable movement of the target on display. It accurately predicts error rate in a range of realistic tasks. Applications include the optimization of difficulty in game-level design.

To press a button, a finger must push down and pull up with the right force and timing. How the motor system succeeds in button-pressing, in spite of neural noise and lacking direct access to the mechanism of the button, is poorly understood. This paper investigates a unifying account based on neuromechanics. Mechanics is used to model muscles controlling the finger that contacts the button. Neurocognitive principles are used to model how the motor system learns appropriate muscle activations over repeated strokes though relying on degraded sensory feedback. Neuromechanical simulations yield a rich set of predictions for kinematics, dynamics, and user performance and may aid in understanding and improving input devices. We present a computational implementation and evaluate predictions for common button types.

As tablet computers are hosting more productivity applications, efficient text entry is becoming more important. A soft keyboard, which is the primary text entry interface for tablets, however, often competes with applications for the limited screen space. A promising solution to this problem may be a semi-transparent soft keyboard (STK), which can share the screen with an application. A few commercial STK products are already available, but research questions about the STK design have not been explored in depth yet. Therefore, we conducted three experiments to answer 1) the effect of the transparency level on usability, 2) exploration of diverse design options for an STK, and 3) the effect of an STK on the different text caret positions. The results imply that STKs with 50% opacity showed a balanced performance; well-designed STKs were acceptable in both content reading and typing situations; which could reach 90-100% of an opaque keyboard in terms of overall performance; and the text caret could intrude the STK down to the number row.

This paper reviews and categorizes the most common types of selfie sticks available in the market and discusses their design and usability. Through a theoretical analysis, it demonstrates how most commercial selfie sticks ignore important human factors, including ergonomics. It, then, presents results of an online survey, where selfie stick users (N=105), predominantly from the Republic of Korea, rated the usability of their selfie sticks. Results of the survey provided an insight into users' selfie stick usage behavior and suggested that most commercial selfie sticks are unergonomic, causing the users short-term fatigues that could "in theory" turn into chronic over the time with extensive use. Finally, based on the survey results, the paper makes recommendations for future design.

Reflector is a novel direct pointing method that utilizes hidden design space on reflective screens. By aligning a part of the user's onscreen reflection with objects rendered on the screen, Reflector enables (1) distance-independent and (2) private pointing on commodity screens. Reflector can be implemented easily in both desktop and mobile conditions through a single camera installed at the edge of the screen. Reflector's pointing performance was compared to today's major direct input devices: eye trackers and touchscreens. We demonstrate that Reflector allows the user to point more reliably, regardless of distance from the screen, compared to an eye tracker. Further, due to the private nature of an onscreen reflection, Reflector shows a shoulder surfing success rate 20 times lower than that of touchscreens for the task of entering a 4-digit PIN.

We present a new smart-restorable backspace technique to facilitate correction of "overlooked" errors on touchscreen-based tablets. We conducted an empirical study to compare the new backspace technique with the conventional one. Results of the study revealed that the new technique improves the overall text entry performance, both in terms of speed and operations per character, by significantly reducing error correction efforts. In addition, results showed that most users preferred the new technique to the one they use on their tablets, and found it easy to learn and use. Most of them also felt that it improved their overall text entry performance, thus wanted to keep using it.

We introduce TapBoard 2, a touchpad-based keyboard that solves the problem of typing and pointing disambiguation. The pointing interaction design of TapBoard 2 is nearly identical to natural touchpad interaction, and its shared workspace naturally invites bimanual pointing interaction. To implement TapBoard 2, we developed a novel gesture representation scheme for a systematic design and gesture recognizer. A user evaluation showed that TapBoard 2 successfully supports collocated pointing and typing interaction. It was able to disambiguate typing and pointing actions with an accuracy of greater than 95%. In addition, the typing and pointing performance of TapBoard 2 were comparable to that of a separate keyboard and mouse. In particular, the bimanual pointing operations of TapBoard 2 are highly efficient and strongly favored by participants.

In this paper, we explore the possibility of a long touchpad that utilizes the entire area below the keyboard of a laptop computer. An essential prerequisite for such a touchpad is a robust palm rejection method, which we satisfy using a proximity-sensing touchpad. We developed LongPad, a proximity-sensing optical touchpad that is as wide as a laptop keyboard, and implemented a palm rejection algorithm that utilizes proximity images from LongPad. In a user study conducted, we observed that LongPad rejected palm touches almost perfectly while participants were repeating typing and pointing tasks. We also summarize the new design space enabled by LongPad and demonstrate a few of the interaction techniques it facilitates.

A physical keyboard key has three states, whereas a touch screen usually has only two. Due to this difference, the state corresponding to the touched state of a physical key is missing in a touch screen keyboard. This touched state is an important factor in the usability of a keyboard. In order to recover the role of a touched state in a touch screen, we propose the TapBoard, a touch screen software keyboard that regards tapping actions as keystrokes and other touches as the touched state. In a series of user studies, we validate the effectiveness of the TapBoard concept. First, we show that tapping to type is in fact compatible with the existing typing skill of most touch screen keyboard users. Second, users quickly adapt to the TapBoard and learn to rest their fingers in the touched state. Finally, we confirm by a controlled experiment that there is no difference in text-entry performance between the TapBoard and a traditional touch screen software keyboard. In addition to these experimental results, we demonstrate a few new interaction techniques that will be made possible by the TapBoard.

In this paper, we present a haptic feedback method for a virtual button based on the force-displacement curves of a physical button. The original feature of the proposed method is that it provides haptic feedback, not only for the "click" sensation but also for the moving sensation before and after transition points in a force-displacement curve. The haptic feedback is by vibrotactile stimulations only and does not require a force feedback mechanism. We conducted user experiments to show that the resultant haptic feedback is realistic and distinctive. Participants were able to distinguish among six different virtual buttons, with 94.1% accuracy even in a noisy environment. In addition, participants were able to associate four virtual buttons with their physical counterparts, with a correct answer rate of 79.2%.