Adaptive interfaces needs to untangle confounded user traits-like preference versus expertise-from sparse behavioral data. We present an Approximate Bayesian Inverse Reinforcement Learning (ABIRL) framework that models users as bounded-rational agents and uses Approximate Bayesian Computation to jointly infer multiple latent parameters. By matching low-dimensional summaries of simulated and observed trajectories, ABIRL recovers posterior distributions over both preference and expertise, rather than collapsing to a single esti-mate. Through a series of controlled grid-world experiments and a human-subject study, we demonstrate reliable disentanglement of these intertwined factors and show that uncertainty-aware intervention policies-grounded in the inferred posteriors-lead to safer, more effective adaptive assistance.

How do we find items within graphical user interfaces (GUIs)? Current understanding of this issue relies on studies using symbol matrices, natural scenes, and other non-GUI stimuli. To understand whether the effects discovered in those environments extend to mobile, desktop, and web interfaces, this paper reports on visual search performance and eye movements with 900 real-world GUIs. In an eye-tracking study, participants (N=84) were given a cue (textual or image) describing a target to find within a GUI. The study found that the type of GUI, the absence/presence of the target, and cue type affected search time more than visual complexity did. We also compared visual search to free-viewing in GUIs, concluding that these two tasks are distinctly different. Synthesis of the results points to a Guess-Scan-Confirm pattern in visual search: in the first few fixations, gaze is frequently directed toward the top-left corner of the screen, a pattern possibly related to the top left being a statistically likely location of the target or of information that could aid in finding it; attention then gets more selectively guided, in line with the GUI’s structure and the features of the target; and, finally, the user must confirm whether the target has been identified or, instead, that no target is visible. The VSGUI10K eye-tracking dataset (10,282 trials) is released for study and modeling of visual search.

Computational models can advance affective science by shedding light onto the interplay between cognition and emotion from an information processing point of view. We propose a computational model of emotion that integrates reinforcement learning (RL) and appraisal theory, establishing a formal relationship between reward processing, goal-directed task learning, cognitive appraisal, and emotional experiences. The model achieves this by formalizing four evaluative checks from the component process model (CPM) in terms of temporal difference learning updates: suddenness, goal relevance, goal conduciveness, and power. The formalism is task independent and can be applied to any task that is represented as a Markov decision problem (MDP) and solved using RL. We evaluate the model by predicting a range of human emotions based on a series of vignette studies, highlighting its potential to improve our understanding of the role of reward processing in affective experiences.

Attention management systems aim to mitigate the negative effects of multitasking. However, sophisticated real-time attention management is yet to be developed. We present a novel concept for attention management with reinforcement learning that automatically switches tasks. The system was trained with a user model based on principles of computational rationality. Due to this user model, the system derives a policy that schedules task switches by considering human constraints such as visual limitations and reaction times. We evaluated its capabilities in a challenging dual-task balancing game. Our results confirm our main hypothesis that an attention management system based on reinforcement learning can significantly improve human performance, compared to humans’ self-determined interruption strategy. The system raised the frequency and difficulty of task switches compared to the users while still yielding a lower subjective workload. We conclude by arguing that the concept can be applied to a great variety of multitasking settings.

Touchscreen typing requires coordinating the fingers and visual attention for button-pressing, proofreading, and error correction. Computational models need to account for the associated fast pace, coordination issues, and closed-loop nature of this control problem, which is further complicated by the immense variety of keyboards and users. The paper introduces CRTypist, which generates human-like typing behavior. Its key feature is a reformulation of the supervisory control problem, with the visual attention and motor system being controlled with reference to a working memory representation tracking the text typed thus far. Movement policy is assumed to asymptotically approach optimal performance in line with cognitive and design-related bounds. This flexible model works directly from pixels, without requiring hand-crafted feature engineering for keyboards. It aligns with human data in terms of movements and performance, covers individual differences, and can generalize to diverse keyboard designs. Though limited to skilled typists, the model generates useful estimates of the typing performance achievable under various conditions.

Predicting users’ emotional states during interaction is a long-standing goal of affective computing. However, traditional methods based on sensory data alone fall short due to the interplay between users’ latent cognitive states and emotional responses. To address this, we introduce a computational cognitive model that simulates emotion as a continuous process, rather than a static state, during interactive episodes. This model integrates cognitive-emotional appraisal mechanisms with computational rationality, utilizing value predictions from reinforcement learning. Experiments with human participants demonstrate the model’s ability to predict and explain the emergence of emotions such as happiness, boredom, and irritation during interactions. Our approach opens the possibility of designing interactive systems that adapt to users’ emotional states, thereby improving user experience and engagement. This work also deepens our understanding of the potential of modeling the relationship between reward processing, reinforcement learning, goal-directed behavior, and appraisal.

It is difficult for interactive systems to adapt to individual users due to the complex relationship between the user's mental model and observable behavior. This paper intro-duces a framework for making parameterized models that build on the assumption of computational rationality to formalize the link between mental models and interactive behavior. With two experiments, we illustrate how to infer the user's mental model from complex and limited observation through iterative inference and update, offering a computationally feasible approach for enhancing the understanding of the user. This work contributes to developing more intuitive and effective interactive systems in HCI and collaborative AI, emphasizing the need for a nuanced understanding of mental models and their impact on interaction.

Understanding the interaction between different road users is critical for road safety and automated vehicles (AVs). Existing mathematical models on this topic have been proposed based mostly on either cognitive or machine learning (ML) approaches. However, current cognitive models are incapable of simulating road user trajectories in general scenarios, and ML models lack a focus on the mechanisms generating the behavior and take a high-level perspective which can cause failures to capture important human-like behaviors. Here, we develop a model of human pedestrian crossing decisions based on computational rationality, an approach using deep reinforcement learning (RL) to learn boundedly optimal behavior policies given human constraints, in our case a model of the limited human visual system. We show that the proposed combined cognitive-RL model captures human-like patterns of gap acceptance and crossing initiation time. Interestingly, our model’s decisions are sensitive to not only the time gap, but also the speed of the approaching vehicle, something which has been described as a “bias” in human gap acceptance behavior. However, our results suggest that this is instead a rational adaption to human perceptual limitations. Moreover, we demonstrate an approach to accounting for individual differences in computational rationality models, by conditioning the RL policy on the parameters of the human constraints. Our results demonstrate the feasibility of generating more human-like road user behavior by combining RL with cognitive models.

How do people interact with computers? This fundamental question was asked by Card, Moran, and Newell in 1983 with a proposition to frame it as a question about human cognition – in other words, as a matter of how information is processed in the mind. Recently, the question has been reframed as one of adaptation: how do people adapt their interaction to the limits imposed by cognition, device design, and environment? The paper synthesizes advances toward an answer within the theoretical framework of computational rationality. The core assumption is that users act in accordance with what is best for them, given the limits imposed by their cognitive architecture and their experience of the task environment. This theory can be expressed in computational models that explain and predict interaction. The paper reviews the theoretical commitments and emerging applications in HCI, and it concludes by outlining a research agenda for future work.

Human factors research and engineering of advanced driving assistance systems (ADAS) must consider how drivers adapt to their presence. The major obstruction to this at the moment is poor understanding of the details of the adaptive processes that the human cognition undergoes when faced with such changes. This paper presents a simulation model that predicts how drivers adapt to a steering assistance system. Our approach is based on computational rationality, and demonstrates how task interleaving strategies adapt to the task environment and the driver’s goals and cognitive limitations. A supervisor controls eye movements between the driving and non-driving tasks, making this choice on the basis of maximising expected joint task utility. The model predicts that with steering assistance, drivers’ in car glance durations increase. We also show that this adaptation leads to risky driving in cases where the reliability of the system is compromised.

Traditionally, touchscreen typing has been studied in terms of motor performance. However, recent research has exposed a decisive role of visual attention being shared between the keyboard and the text area. Strategies for this are known to adapt to the task, design, and user. In this paper, we propose a unifying account of touchscreen typing, regarding it as optimal supervisory control. Under this theory, rules for controlling visuo-motor resources are learned via exploration in pursuit of maximal typing performance. The paper outlines the control problem and explains how visual and motor limitations affect it. We then present a model, implemented via reinforcement learning, that simulates co-ordination of eye and finger movements. Comparison with human data affirms that the model creates realistic finger- and eye-movement patterns and shows human-like adaptation. We demonstrate the model’s utility for interface development in evaluating touchscreen keyboard designs.

Over the past decade, responsive web design (RWD) has become the de facto standard for adapting web pages to a wide range of devices used for browsing. While RWD has improved the usability of web pages, it is not without drawbacks and limitations: designers and developers must manually design the web layouts for multiple screen sizes and implement associated adaptation rules, and its "one responsive design fits all" approach lacks support for personalization. This paper presents a novel approach for automated generation of responsive and personalized web layouts. Given an existing web page design and preferences related to design objectives, our integer programming -based optimizer generates a consistent set of web designs. Where relevant data is available, these can be further automatically personalized for the user and browsing device. The paper includes presentation of techniques for runtime adaptation of the designs generated into a fully responsive grid layout for web browsing. Results from our ratings-based online studies with end users (N = 86) and designers (N = 64) show that the proposed approach can automatically create high-quality responsive web layouts for a variety of real-world websites.

Relatively little is known about eye and finger movement in typing with mobile devices. Most prior studies of mobile typing rely on log data, while data on finger and eye movements in typing come from studies with physical keyboards. This paper presents new findings from a transcription task with mobile touchscreen devices. Movement strategies were found to emerge in response to sharing of visual attention: attention is needed for guiding finger movements and detecting typing errors. In contrast to typing on physical keyboards, visual attention is kept mostly on the virtual keyboard, and glances at the text display are associated with performance. When typing with two fingers, although users make more errors, they manage to detect and correct them more quickly. This explains part of the known superiority of two-thumb typing over one-finger typing. We release the extensive dataset on everyday typing on smartphones.

A central concern in an interactive intelligent system is optimization of its actions, to be maximally helpful to its human user. In recommender systems for instance, the action is to choose what to recommend, and the optimization task is to recommend items the user prefers. The optimization is done based on earlier user's feedback (e.g. "likes" and "dislikes"), and the algorithms assume the feedback to be faithful. That is, when the user clicks "like," they actually prefer the item. We argue that this fundamental assumption can be extensively violated by human users, who are not passive feedback sources. Instead, they are in control, actively steering the system towards their goal. To verify this hypothesis, that humans steer and are able to improve performance by steering, we designed a function optimization task where a human and an optimization algorithm collaborate to find the maximum of a 1-dimensional function. At each iteration, the optimization algorithm queries the user for the value of a hidden function f at a point x, and the user, who sees the hidden function, provides an answer about f(x). Our study on 21 participants shows that users who understand how the optimization works, strategically provide biased answers (answers not equal to f(x)), which results in the algorithm finding the optimum significantly faster. Our work highlights that next-generation intelligent systems will need user models capable of helping users who steer systems to pursue their goals.

We present a computational model of visual search on graphical layouts. It assumes that the visual system is maximising expected utility when choosing where to fixate next. Three utility estimates are available for each visual search target: one by unguided perception only, and two, where perception is guided by long-term memory (location or visual feature). The system is adaptive, starting to rely more upon long-term memory when its estimates improve with experience. However, it needs to relapse back to perception-guided search if the layout changes. The model provides a tool for practitioners to evaluate how easy it is to find an item for a novice or an expert, and what happens if a layout is changed. The model suggests, for example, that (1) layouts that are visually homogeneous are harder to learn and more vulnerable to changes, (2) elements that are visually salient are easier to search and more robust to changes, and (3) moving a non-salient element far away from original location is particularly damaging. The model provided a good match with human data in a study with realistic graphical layouts.

In domains where users are exposed to large variations in visuo-spatial features among designs, they often spend excess time searching for common elements (features) on an interface. This article contributes individualised predictive models of visual search, and a computational approach to restructure graphical layouts for an individual user such that features on a new, unvisited interface can be found quicker. It explores four technical principles inspired by the human visual system (HVS) to predict expected positions of features and create individualised layout templates: (I) the interface with highest frequency is chosen as the template; (II) the interface with highest predicted recall probability (serial position curve) is chosen as the template; (III) the most probable locations for features across interfaces are chosen (visual statistical learning) to generate the template; (IV) based on a generative cognitive model, the most likely visual search locations for features are chosen (visual sampling modelling) to generate the template. Given a history of previously seen interfaces, we restructure the spatial layout of a new (unseen) interface with the goal of making its features more easily findable. The four HVS principles are implemented in Familiariser, a web browser that automatically restructures webpage layouts based on the visual history of the user. Evaluation of Familiariser (using visual statistical learning) with users provides first evidence that our approach reduces visual search time by over 10%, and number of eye-gaze fixations by over 20%, during web browsing tasks.

In domains where users are exposed to large variations in visuo-spatial features among designs, they often spend excess time searching for common elements (features) in familiar locations. This paper contributes computational approaches to restructuring layouts such that features on a new, unvisited interface can be found quicker. We explore four concepts of familiarisation, inspired by the human visual system (HVS), to automatically generate a familiar design for each user.

This article examines the cognitive process of visually experiencing user interfaces. It contributes to a theory- and methodology-grounded understanding of how UIs are experienced with regard to various aesthetic criteria. This aids in considering the targeted experience goals in relation to visual design choices—a problem that designers usually have to tackle intuitively. The issue in explicitly relating designs to experiences stems from the complexity of the process in which visual stimuli are processed and turned into experiences. The authors present a cognitive top-down approach to this process, rooted in the appraisal theory and the theory of the predictive brain. Several predictions are derived via this approach, and an eye-tracking experiment with Web sites is presented that provides evidence of them. The experience goals and repeated exposure to stimuli are shown to affect appraisal times and visual scanpaths in Web pages’ evaluation; this supports the top-down approach described. Researchers can use the findings to inform their theoretical and empirical pursuits as they strive to understand what makes design artefacts emotionally evocative, and the methodology outlined can assist designers in locating the visual regions and elements relevant for experiential design goals.

Predicting how users learn new or changed interfaces is a long-standing objective in HCI research. This paper contributes to understanding of visual search and learning in text entry. With a goal of explaining variance in novices' typing performance that is attributable to visual search, a model was designed to predict how users learn to locate keys on a keyboard: initially relying on visual short-term memory but then transitioning to recall-based search. This allows predicting search times and visual search patterns for completely and partially new layouts. The model complements models of motor performance and learning in text entry by predicting change in visual search patterns over time. Practitioners can use it for estimating how long it takes to reach the desired level of performance with a given layout.

The core idea of experience-driven design is to define the intended experience before functionality and technology. This is a radical idea for companies that have built their competences around specific technologies. Although many technology companies are willing to shift their focus towards experience-driven design, reports on real-life cases about the utilization of this design approach are rare. As part of an industry-led research program, we introduced experience-driven design to metal industry companies with experience goals as the key technique. Four design cases in three companies showed that the goals are useful in keeping the focus on user experience, but several challenges are still left for future research to tackle. This exploratory research lays ground for future research by providing initial criteria for assessing experience design tools. The results shed light on utilizing experience goals in industrial design projects and help practitioners in planning and managing the product design process with user experience in mind.

Technological artefacts express time periods in their visual design. Due time, visual culture changes and thus affects the design of pictorial representations in technological products, such as icons in user interfaces. Previous research of temporal aspects in human-computer interaction has been focusing on particular interaction situations, but not on the effects of design eras on user experience. The influence of icon design styles of different eras on aesthetic and usability experiences was studied with the method of primed product comparisons. Affective preferences and their processing times were analysed in order to examine visual usability in terms of semantic distance and aesthetic appeal of icons from different design eras. Aesthetic and usability preferences of icons from different eras varied, which allowed the investigation of the process in which users experience icons. This examination results in elaborating the process, for example the relationship between cognitive processing fluency, familiarity, and beauty.

A method for primed product comparisons was developed, based on the methodological considerations of emotional appraisal process and affective mental contents. The method was implemented as a computer tool, which was utilised in two experiments (N = 18 for both). Ten adjectives served as primes, and five drinking glass pictures as stimuli. Participants' task was to choose a preference between two glasses, given the priming adjective. The results validate the method by providing test-retest reliability measures and showing convergence with questionnaires. Further, different evaluation times between the primes and the stimuli reveal the existence of different mental processes associated with various aspects of product experience, as predicted by appraisal theory. The results have various implications for experience research and development in HCI, as they demonstrate how the method can be used for product evaluation and the analysis of the mental processes, which users use to evaluate the products.

The key prerequisite for experience-driven design is to define what experience to design for. User experience (UX) goals concretise the intended experience. Based on our own case studies from industrial environments and a literature study, we propose five different approaches to acquiring insight and inspiration for UX goal setting: Brand, Theory, Empathy, Technology, and Vision. Each approach brings in a different viewpoint, thus supporting the multidisciplinary character of UX. The Brand approach ensures that the UX goals are in line with the company's brand promise. The Theory approach utilises the available scientific knowledge of human behaviour. The Empathy approach focuses on knowing the actual users and stepping into their shoes. The Technology approach considers the new technologies that are being introduced and their positive or negative influence on UX. Finally, the Vision approach focuses on renewal, introducing new kinds of UXs. In the design of industrial systems, several stakeholders are involved and they should share common design goals. Using the different UX goal-setting approaches together brings in the viewpoints of different stakeholders, thus committing them to UX goal setting and emphasising UX as a strategic design decision.

Open educational resources (OERs) provide opportunities as enablers of societal development, but they also create new challenges. From the perspective of content providers and educational institutions, particularly, cultural and context-related challenges emerge. Even though barriers regarding large-scale adoption of OERs are widely discussed, empirical evidence for determining challenges in relation to particular contexts is still rare. Such context-specific barriers generally can jeopardize the acceptance of OERs and, in particular, social OER environments. We conducted a large-scale (N = 855) cross-European investigation in the school context to determine how teachers and learners perceive cultural distance as a barrier against the use of social OER environments. The findings indicate how nationality and age of the respondents are strong predictors of cultural distance barrier. The study concludes with identification of context-sensitive interventions for overcoming the related barriers. These consequences are vital for OER initiatives and educational institutions for aligning their efforts on OER.

User psychology is a human–technology interaction research approach that uses psychological concepts, theories, and findings to structure problems of human–technology interaction. As the notion of user experience has become central in human–technology interaction research and in product development, it is necessary to investigate the user psychology of user experience. This analysis of emotional human–technology interaction is based on the psychological theory of basic emotions. Three studies, two laboratory experiments, and one field study are used to investigate the basic emotions and the emotional mind involved in user experience. The first and second experiments study the measurement of subjective emotional experiences during novel human–technology interaction scenarios in a laboratory setting. The third study explores these aspects in a real-world environment. As a result of these experiments, a bipolar competence–frustration model is proposed, which can be used to understand the emotional aspects of user experience.