Locating a target based on auditory and visual cues—such as finding a car in a crowded parking lot or identifying a speaker in a virtual meeting—requires balancing effort, time, and accuracy under uncertainty. Existing models of audiovisual search often treat perception and action in isolation, overlooking how people adaptively coordinate movement and sensory strategies. We present Sensonaut, a computational model of embodied audiovisual search. The core assumption is that people deploy their body and sensory systems in ways they believe will most efficiently improve their chances of locating a target, trading off time and effort under perceptual constraints. Our model formulates this as a resource-rational decision-making problem under partial observability. We validate the model against newly collected human data, showing that it reproduces both adaptive scaling of search time and effort under task complexity, occlusion, and distraction, and characteristic human errors. Our simulation of human-like resource-rational search informs the design of audiovisual interfaces that minimize search cost and cognitive load.

The rise in popularity and value of esports motivates the creation of computational training tools (CTTs) for learning, assessment, and skill gain. While some tools exist commercially, much of the work in the research literature is rarely used outside of a lab, resulting in a lack of knowledge on the challenges involved in real-world integration. In this work, we develop a bespoke CTT for League of Legends—MySkills—based on prior work and deploy it at a professional training academy for three months. Based on two rounds of stakeholder interviews, we uncover insights into users’ perspectives on using CTTs in esports coaching and the challenges inherent in introducing a novel tool into an existing, real-world esports training context. From these results, we connect the domain of esports training technology to existing conversations on translational HCI, challenges in bridging research and practice, and present implications for future work.

Computer mice are widely used today as the primary input device in competitive video games. If a player exhibits more wrist rotation than other players when moving the mouse laterally, the player is said to have stronger wrist-aiming habits. Despite strong public interest, there has been no affordable technique to quantify the extent of a player’s wrist-aiming habits and no scientific investigation into how the habits affect player performance and workload. We present a reliable and affordable technique to quantify the extent of a player’s wrist-aiming habits using a mouse equipped with two optical sensors (i.e., a dual-sensor mouse). In two user studies, we demonstrate the reliability of the technique and examine the relationship between wrist-aiming habits and player performance or workload. In summary, player expertise and mouse sensitivity significantly impacted wrist-aiming habits; the extent of wrist-aiming showed a positive correlation with upper limb workload.

A significant amount of research has recently been conducted on user performance in so-called temporal pointing tasks, in which a user is required to perform a button input at the timing required by the system. Consecutive temporal pointing (CTP), in which two consecutive button inputs must be performed while satisfying temporal constraints, is common in modern interactions, yet little is understood about user performance on the task. Through a user study involving 100 participants, we broadly explore user performance in a variety of CTP scenarios. The key finding is that CTP is a unique task that cannot be considered as two ordinary temporal pointing processes. Significant effects of button input method, motor limitations, and different hand use were also observed.

To achieve high performance in esports, players must be able to effectively and efficiently control input devices such as a computer mouse and keyboard (i.e., input skills). Characterizing and quantifying a player’s input skills can provide useful insights, but collecting and analyzing sufficient amounts of data in ecologically valid settings remains a challenge. Targeting the popular esports game, League of Legends, we go beyond the limitations of previous studies and demonstrate a holistic pipeline of input behavior analysis: from quantifying the quality of players’ input behavior (i.e., input skill) to training players based on the analysis. Based on interviews with five top-tier professionals and analysis of input behavior logs from 4,835 matches played freely at home collected from 193 players (including 18 professionals), we confirmed that players with higher ranks in the game implement eight different input skills with higher quality. In a three-week follow-up study using a training aid that visualizes a player’s input skill levels, we found that the analysis provided players with actionable lessons, potentially leading to meaningful changes in their input behavior.

Selecting a target in a 3D environment is often challenging, especially with small/distant targets or when sensor noise is high. To facilitate selection, target-inference methods must be accurate, fast, and account for noise and motor variability. However, traditional data-free approaches fall short in accuracy since they ignore variability. While data-driven solutions achieve higher accuracy, they rely on extensive human datasets so prove costly, time-consuming, and transfer poorly. In this paper, we propose a novel approach that leverages biomechanical simulation to produce synthetic motion data, capturing a variety of movement-related factors, such as limb configurations and motor noise. Then, an inference model is trained with only the simulated data. Our simulation-based approach improves transfer and lowers cost; variety-rich data can be produced in large quantities for different scenarios. We empirically demonstrate that our method matches the accuracy of human-data-driven approaches using data from seven users. When deployed, the method accurately infers intended targets in challenging 3D pointing conditions within 5–10 milliseconds, reducing users’ target-selection error by 71% and completion time by 35%.

There have been significant advances in simulation models predicting human behavior across various interactive tasks. One issue remains, however: identifying the parameter values that best describe an individual user. These parameters often express personal cognitive and physiological characteristics, and inferring their exact values has significant effects on individual-level predictions. Still, the high complexity of simulation models usually causes parameter inference to consume prohibitively large amounts of time, as much as days per user. We investigated amortized inference for its potential to reduce inference time dramatically, to mere tens of milliseconds. Its principle is to pre-train a neural proxy model for probabilistic inference, using synthetic data simulated from a range of parameter combinations. From examining the efficiency and prediction performance of amortized inference in three challenging cases that involve real-world data (menu search, point-and-click, and touchscreen typing), the paper demonstrates that an amortized-inference approach permits analyzing large-scale datasets by means of simulation models. It also addresses emerging opportunities and challenges in applying amortized inference in HCI.

In this study, we analyzed how the performance of professional Go players has changed since the advent of AlphaGo, the first artificial intelligence (AI) application to defeat a human world Go champion. We interviewed and surveyed professional Go players and found that AI has been actively introduced into the Go training process since the advent of AlphaGo. The significant impact of AI-based training was confirmed in a subsequent analysis of 6,292 games in Korean Go tournaments and Elo rating data of 1,362 Go players worldwide. Overall, the tendency of players to make moves similar to those recommended by AI has sharply increased since 2017. The degree to which players’ expected win rates fluctuate during a game has also decreased significantly since 2017. We also found that AI-based training has provided more benefits to senior players and allowed them to achieve Elo ratings higher than those of junior players.

When giving input with a button, users follow one of two strategies: (1) react to the output from the computer or (2) proactively act in anticipation of the output from the computer. We propose a technique to quantify reactiveness and proactiveness to determine the degree and characteristics of each input strategy. The technique proposed in this study uses only screen recordings and does not require instrumentation beyond the input logs. The likelihood distribution of the time interval between the button inputs and system outputs, which is uniquely determined for each input strategy, is modeled. Then the probability that each observed input/output pair originates from a specific strategy is estimated along with the parameters of the corresponding likelihood distribution. In two empirical studies, we show how to use the technique to answer questions such as how to design animated transitions and how to predict a player’s score in real-time games.

The simulation of user behavior with deep reinforcement learning agents has shown some recent success. However, the inverse problem, that is, inferring the free parameters of the simulator from observed user behaviors, remains challenging to solve. This is because the optimization of the new action policy of the simulated agent, which is required whenever the model parameters change, is computationally impractical. In this study, we introduce a network modulation technique that can obtain a generalized policy that immediately adapts to the given model parameters. Further, we demonstrate that the proposed technique improves the efficiency of user simulator-based inference by eliminating the need to obtain an action policy for novel model parameters. We validated our approach using the latest user simulator for point-and-click behavior. Consequently, we succeeded in inferring the user’s cognitive parameters and intrinsic reward settings with less than 1/1000 computational power to those of existing methods.

We present a novel simulation model of point-and-click behaviour that is applicable both when a target is stationary or moving. To enable more realistic simulation than existing models, the model proposed in this study takes into account key features of the user and the external environment, such as intermittent motor control, click decision-making, visual perception, upper limb kinematics and the effect of input device. The simulated user’s point-and-click behaviour is formulated as a Markov decision process (MDP), and the user’s policy of action is optimised through deep reinforcement learning. As a result, our model successfully and accurately reproduced the trial completion time, distribution of click endpoints, and cursor trajectories of real users. Through an ablation study, we showed how the simulation results change when the model’s sub-modules are individually removed. The implemented model and dataset are publicly available.

In first-person shooters (FPS), professional players (a.k.a., Gosu) outperform amateur players. The secrets behind the performance of professional FPS players have been debated in online communities with many conjectures; however, attempts of scientific verification have been limited. We addressed this conundrum through a data-collection study of the gameplay of eight professional and eight amateur players in the commercial FPS Counter-Strike: Global Offensive. The collected data cover behavioral data from six sensors (motion capture, eye tracker, mouse, keyboard, electromyography armband, and pulse sensor) and in-game data (player data and event logs). We examined conjectures in four categories: aiming, character movement, physicality, and device and settings. Only 6 out of 13 conjectures were supported with statistically sufficient evidence.

Playtesting is widely performed in the game industry to gauge the difficulty of a game. A large number of test participants with different skills must be recruited for reliable test results, resulting in high costs. Automated playtesting based on player simulation is expected to reduce playtesting costs. Still, it has not yet been widely applied due to the lack of a method that realistically simulates players' gameplays with different skills. Based on a cognitive model of sensorimotor coordination that explains the human button input process, we propose a novel automated playtesting technique that predicts the game difficulty experienced by players with different skills in moving-target acquisition (MTA) games. The model has free parameters representing the inherent skills of players. Once the parameters are obtained for a specific population (e.g., seniors), it is possible to estimate the game difficulty at the population level in multiple games. We applied the technique to two simple MTA games and showed that it could predict the relative difference in game difficulties experienced by players with different skills.

Designing a push-button with desired sensation and performance is challenging because the mechanical construction must have the right response characteristics. Physical simulation of a button's force-displacement (FD) response has been studied to facilitate prototyping; however, the simulations' scope and realism have been limited. In this paper, we extend FD modeling to include vibration (V) and velocity-dependence characteristics (V). The resulting FDVV models better capture tactility characteristics of buttons, including snap. They increase the range of simulated buttons and the perceived realism relative to FD models. The paper also demonstrates methods for obtaining these models, editing them, and simulating accordingly. This end-to-end approach enables the analysis, prototyping, and optimization of buttons, and supports exploring designs that would be hard to implement mechanically.

In virtual reality (VR), a user's virtual avatar can interact with a virtual object by colliding with it. If collision responses do not occur in the direction that the user expects, the user experiences degradation of accuracy and precision in applications such as VR sports games. In determining the response of a virtual collision, existing physics engines have not considered the direction in which the user perceived and estimated the collision. Based on the cue integration theory, this study presents a statistical model explaining how users estimate the direction of a virtual collision from their body's orientation and velocity vectors. The accuracy and precision of virtual collisions can be improved by 8.77% and 30.29%, respectively, by setting the virtual collision response in the direction that users perceive.

Computer mice have their displacement sensors in various locations (center, front, and rear). However, there has been little research into the effects of sensor position or on engineering approaches to exploit it. This paper first discusses the mechanisms via which sensor position affects mouse movement and reports the results from a study of a pointing task in which the sensor position was systematically varied. Placing the sensor in the center turned out to be the best compromise: improvements over front and rear were in the 11-14% range for throughput and 20--23% for path deviation. However, users varied in their personal optima. Accordingly, variable-sensor-position mice are then presented, with a demonstration that high accuracy can be achieved with two static optical sensors. A virtual sensor model is described that allows software-side repositioning of the sensor. Individual-specific calibration should yield an added 4% improvement in throughput over the default center position.

A well-designed control-to-display gain function can improve pointing performance with indirect pointing devices like trackpads. However, the design of gain functions is challenging and mostly based on trial and error. AutoGain is a novel method to individualize a gain function for indirect pointing devices in contexts where cursor trajectories can be tracked. It gradually improves pointing efficiency by using a novel submovement-level tracking+optimization technique that minimizes aiming error (undershooting/overshooting) for each submovement. We first show that AutoGain can produce, from scratch, gain functions with performance comparable to commercial designs, in less than a half-hour of active use. Second, we demonstrate AutoGain's applicability to emerging input devices (here, a Leap Motion controller) with no reference gain functions. Third, a one-month longitudinal study of normal computer use with AutoGain showed performance improvements from participants' default functions.

Pointing is the task of tracking a target with a pointer and confirming the target selection through a click action when the pointer is positioned within the target. Little is known about the mechanism by which users plan and execute the click action in the middle of the target tracking process. The Intermittent Click Planning model proposed in this study describes the process by which users plan and execute optimal click actions, from which the model predicts the pointing error rates. In two studies in which users pointed to a stationary target and a moving target, the model proved to accurately predict the pointing error rates (R2 = 0.992 and 0.985, respectively). The model has also successfully identified differences in cognitive characteristics among first-person shooter game players.

Effects of unintended latency on gamer performance have been reported. End-to-end latency can be corrected by post-input manipulation of activation times, but this gives the player unnatural gameplay experience. For moving-target selection games such as Flappy Bird, the paper presents a predictive model of latency on error rate and a novel compensation method for the latency effects by adjusting the game's geometry design -- e.g., by modifying the size of the selection region. Without manipulation of the game clock, this can keep the user's error rate constant even if the end-to-end latency of the system changes. The approach extends the current model of moving-target selection with two additional assumptions about the effects of latency: (1) latency reduces players' cue-viewing time and (2) pushes the mean of the input distribution backward. The model and method proposed have been validated through precise experiments.

Force feedback is said to be the next frontier in virtual reality (VR). Recently, with consumers pushing forward with untethered VR, researchers turned away from solutions based on bulky hardware (e.g., exoskeletons and robotic arms) and started exploring smaller portable or wearable devices. However, when it comes to rendering inertial forces, such as when moving a heavy object around or when interacting with objects with unique mass properties, current ungrounded force feedback devices are unable to provide quick weight shifting sensations that can realistically simulate weight changes over 2D surfaces. In this paper we introduce Aero-plane, a force-feedback handheld controller based on two miniature jet propellers that can render shifting weights of up to 14 N within 0.3 seconds. Through two user studies we: (1) characterize the users' ability to perceive and correctly recognize different motion paths on a virtual plane while using our device; and, (2) tested the level of realism and immersion of the controller when used in two VR applications (a rolling ball on a plane, and using kitchen tools of different shapes and sizes). Lastly, we present a set of applications that further explore different usage cases and alternative form-factors for our device.

In film directing, a bias towards the representation of a particular gender can cause the audience to form a distorted stereotype of the gender role. The Bechdel test has been widely used to objectively judge the existence of such bias in films. However, because its analysis is based solely on the script of a film, the Bechdel test is incapable of considering the broad spectrum of bias that films can have as a visual medium. This study proposes a more comprehensive analysis system that quantifies the degree of bias in the visual representations of female and male characters in commercial films. By analyzing the image frames of a movie using the latest image analysis techniques, a total of 40 films were analyzed based on 8 quantitative indices. The result demonstrates that there exists a statistically significant difference in the visual representation of female and male characters. Specifically, female characters showed lower values in emotional diversity, spatial occupancy, and temporal occupancy compared to male characters in commercial films. Further, female characters were less likely to wear eyeglasses and also appeared more in static scenes, such as indoors.

The activation point of a button is defined as the depth at which it invokes a make signal. Regular buttons are activated during the downward stroke, which occurs within the first 20 ms of a press. The remaining portion, which can be as long as 80 ms, has not been examined for button activation for reason of mechanical limitations. The paper presents a technique and empirical evidence for an activation technique called Impact Activation, where the button is activated at its maximal impact point. We argue that this technique is advantageous particularly in rapid, repetitive button pressing, which is common in gaming and music applications. We report on a study of rapid button pressing, wherein users' timing accuracy improved significantly with use of Impact Activation. The technique can be implemented for modern push-buttons and capacitive sensors that generate a continuous signal.

This paper investigates a common task requiring temporal precision: the selection of a rapidly moving target on display by invoking an input event when it is within some selection window. Previous work has explored the relationship between accuracy and precision in this task, but the role of visual cues available to users has remained unexplained. To expand modeling of timing performance to multimodal settings, common in gaming and music, our model builds on the principle of probabilistic cue integration. Maximum likelihood estimation (MLE) is used to model how different types of cues are integrated into a reliable estimate of the temporal task. The model deals with temporal structure (repetition, rhythm) and the perceivable movement of the target on display. It accurately predicts error rate in a range of realistic tasks. Applications include the optimization of difficulty in game-level design.

To press a button, a finger must push down and pull up with the right force and timing. How the motor system succeeds in button-pressing, in spite of neural noise and lacking direct access to the mechanism of the button, is poorly understood. This paper investigates a unifying account based on neuromechanics. Mechanics is used to model muscles controlling the finger that contacts the button. Neurocognitive principles are used to model how the motor system learns appropriate muscle activations over repeated strokes though relying on degraded sensory feedback. Neuromechanical simulations yield a rich set of predictions for kinematics, dynamics, and user performance and may aid in understanding and improving input devices. We present a computational implementation and evaluate predictions for common button types.

We make touch input by physically colliding an end effector (e.g., a body part or a stylus) with a touch surface. Prior studies have examined the use of kinematic variables of collision between objects, such as position, velocity, force, and impact. However, the nature of the collision can be understood more thoroughly by considering the known physical relationships that exist between directly measurable variables (i.e., kinetics). Based on this collision kinetics, this study proposes a novel touch technique called FDSense. By simultaneously observing the force and contact area measured from the touchpad, FDSense allows estimation of the Young's modulus and stiffness of the object being contacted. Our technical evaluation showed that FDSense could effectively estimate the Young's modulus of end effectors made of various materials, and the stiffness of each part of the human hand. Two applications using FDSense were demonstrated, for digital painting and digital instruments, where the result of the expression varies significantly depending on the elasticity of the end effector. In a following informal study, participants assessed the technique positively.

Virtual collision techniques are interaction techniques for invoking discrete events in a virtual scene, e.g. throwing, pushing, or pulling an object with a pointer. The conventional approach involves detecting collisions as soon as the pointer makes contact with the object. Furthermore, in general, motor patterns can only be adjusted based on visual feedback. The paper presents a multimodal technique based on the principle that collisions should be aligned with the most salient sensory feedback. Boxer (1) triggers a collision at the moment where the pointer's speed reaches a minimum after first contact and (2) is synchronized with vibrotactile stimuli presented to the hand controlling the pointer. Boxer was compared with the conventional technique in two user studies (with temporal pointing and virtual batting). Boxer improved spatial precision in collisions by 26.7 % while accuracy was compromised under some task conditions. No difference was found in temporal precision. Possibilities for improving virtual collision techniques are discussed.

Reflector is a novel direct pointing method that utilizes hidden design space on reflective screens. By aligning a part of the user's onscreen reflection with objects rendered on the screen, Reflector enables (1) distance-independent and (2) private pointing on commodity screens. Reflector can be implemented easily in both desktop and mobile conditions through a single camera installed at the edge of the screen. Reflector's pointing performance was compared to today's major direct input devices: eye trackers and touchscreens. We demonstrate that Reflector allows the user to point more reliably, regardless of distance from the screen, compared to an eye tracker. Further, due to the private nature of an onscreen reflection, Reflector shows a shoulder surfing success rate 20 times lower than that of touchscreens for the task of entering a 4-digit PIN.

We present a novel model to predict error rates in temporal pointing. With temporal pointing, a target is about to appear within a limited time window for selection. Unlike in spatial pointing, there is no movement to control in the temporal domain; the user can only determine when to launch the response. Although this task is common in interactions requiring temporal precision, rhythm, or synchrony, no previous HCI model predicts error rates as a function of task properties. Our model assumes that users have an implicit point of aim but their ability to elicit the input event at that time is hampered by variability in three processes: 1) an internal time-keeping process, 2) a response-execution stage, and 3) input processing in the computer. We derive a mathematical model with two parameters from these assumptions. High fit is shown for user performance with two task types, including a rapidly paced game. The model can explain previous findings showing that touchscreens are much worse in temporal pointing than physical input devices. It also has novel implications for design that extend beyond the conventional wisdom of minimising latency.

The paper contributes a novel technique that can improve user performance in skim reading. Users typically use a continuous-rate-based scrolling technique to skim works such as longer Web pages, e-books, and PDF files. However, visual attention is compromised at higher scrolling rates because of motion blur and extraneous objects with overly brief exposure times. In response, we present Spotlights. It complements the regular continuous technique at high speeds (2--20 pages/s). We present a novel design rule informed by theories of the human visual system for dynamically selecting objects and placing them on transparent overlays on top of the viewer. This improves the quality of visual processing at high scrolling rates by 1) limiting the number of objects, 2) ensuring minimal processing time per object, and 3) keeping objects static to avoid motion blur and facilitate gaze deployment. Spotlights was compared to continuous scrolling in two studies using long documents (200+ pages). Comprehension levels for long documents were comparable with those in continuous-rate-based scrolling, but Spotlights showed significantly better scrolling speed, gaze deployment, recall, lookup performance, and user-rated comprehension.

The computer mouse is rarely used for drawing due to its body-fixed coordinate system, which creates a stroke that differs from the user’s original hand movement. In this study, we resolve this problem by implementing a new mouse called StereoMouse, which eliminates the rotational disturbance of the coordinate system in real-time. StereoMouse is a special mouse with two optical sensors, and its coordinate orientation at the beginning of a stroke is maintained throughout the movement by measuring and compensating for the angular deviation estimated from those sensors. The drawing performance of StereoMouse was measured by means of having users perform the task of repeatedly drawing a basic shape. The results of this experiment showed that StereoMouse eliminated the horizontal drift typically observed in a stroke drawn by a normal mouse. Consequently, StereoMouse allowed the users to draw shapes at a 10.6% faster mean speed with a 10.4% shorter travel time than a normal mouse would. Furthermore, StereoMouse showed 37.1% lower chance of making incorrect gesture input than the normal mouse.

Both of high-rate and low-rate wireless personal area networks (WPANs), being under development by several standardization bodies, are considering to adopt distributed TDMA MAC protocols in order to provide seamless user mobility for WPAN mesh as well as a good peer-to-peer QoS. It has been, however, pointed out that the absence of a central controller in the wireless TDMA MAC may cause a severe performance degradation: e.g., fair allocation, service differentiation, and admission control may be hard to achieve or can not be provided. In this paper, we suggest a new framework of resource allocation for the distributed MAC protocols in WPANs. Simulation results show that our algorithm achieves both a fair resource allocation and flexible service differentiations in a fully distributed way for mesh WPANs where the devices have high mobility and various requirements. We also provide an analytical modeling to discuss about its unique equilibrium and to compute the lengths of reserved time slots at the stable point.