Visible and infrared image fusion (VIF) has attracted significant attention in recent years. Traditional VIF methods primarily focus on generating fused images with high visual quality, while recent advancements increasingly emphasize incorporating semantic information into the fusion model during training. However, most existing segmentation-oriented VIF methods adopt a cascade structure comprising separate fusion and segmentation models, leading to increased network complexity and redundancy. This raises a critical question: can we design a more concise and efficient structure to integrate semantic information directly into the fusion model during training? Inspired by multi-task learning (MTL), we propose a concise and universal training framework, MultiTaskVIF, for segmentation-oriented VIF models. In this framework, we introduce a multi-task head decoder (MTH) that leverages the segmentation head to inject informative semantics into the fusion branch during training. Unlike previous cascade training frameworks that necessitate joint training with a complete segmentation model, MultiTaskVIF enables the existing image fusion model to learn semantic features by simply replacing its decoder with the proposed MTH. Moreover, by combining a hard parameter sharing MTL architecture with an attention mechanism, our method not only reduces network complexity and redundancy but also enhances the fusion model's ability to extract segmentation-relevant features, thereby improving the quality of the fused image for downstream segmentation tasks. Extensive experimental evaluations validate the effectiveness of the proposed method. Our code is publicly available at https://github.com/CnoyZ/MultiTaskVIF.

In dyadic interactions, a broad spectrum of human facial reactions might be appropriate for responding to each human speaker behaviour. Following the successful organisation of the REACT 2023 and REACT 2024 challenges, we are proposing the REACT 2025 challenge encouraging the development and benchmarking of Machine Learning (ML) models that can be used to generate multiple appropriate, diverse, realistic and synchronised human-style facial reactions expressed by human listeners in response to an input stimulus (i.e., audio-visual behaviours expressed by their corresponding speakers). As a key of the challenge, we provide challenge participants with the first natural and large-scale multi-modal Multiple Appropriate Facial Reaction Generation (MAFRG) dataset (called MARS) recording 136 human-human dyadic interactions containing a total of 2856 interaction sessions covering five different topics. In addition, this paper also presents the challenge guidelines and the performance of our baselines on the two proposed sub-challenges: Offline MAFRG and Online MAFRG, respectively. The challenge baseline code is publicly available at https://github.com/reactmultimodalchallenge/baseline_react2025

Preference learning methods make use of models of human choice in order to infer the latent utilities that underlie human behavior. However, accurate modeling of human choice behavior is challenging due to a range of context effects that arise from how humans contrast and evaluate options. Cognitive science has proposed several models that capture these intricacies but, due to their intractable nature, work on preference learning has, in practice, had to rely on tractable but simplified variants of the well-known Bradley-Terry model. In this paper, we take one state-of-the-art intractable cognitive model and propose a tractable surrogate that is suitable for deployment in preference learning. We then introduce a mechanism for fitting the surrogate to human data and extend it to account for data that cannot be explained by the original cognitive model. We demonstrate on large-scale human data that this model produces significantly better inferences on static and actively elicited data than existing Bradley-Terry variants. We further show in simulation that when using this model for preference learning, we can significantly improve utility in a range of real-world tasks.

User models play an important role in interaction design, supporting automation of interaction design choices. In order to do so, model parameters must be estimated from user data. While very large amounts of user data are sometimes required, recent research has shown how experiments can be designed so as to gather data and infer parameters as efficiently as possible, thereby minimising the data requirement. In the current article, we investigate a variant of these methods that amortises the computational cost of designing experiments by training a policy for choosing experimental designs with simulated participants. Our solution learns which experiments provide the most useful data for parameter estimation by interacting with in-silico agents sampled from the model space thereby using synthetic data rather than vast amounts of human data. The approach is demonstrated for three progressively complex models of pointing.

Affordance refers to the perception of possible actions allowed by an object. Despite its relevance to human–computer interaction, no existing theory explains the mechanisms that underpin affordance-formation; that is, how affordances are discovered and adapted via interaction. We propose an integrative theory of affordance-formation based on the theory of reinforcement learning in cognitive sciences. The key assumption is that users learn to associate promising motor actions to percepts via experience when reinforcement signals (success/failure) are present. They also learn to categorize actions (e.g., “rotating” a dial), giving them the ability to name and reason about affordance. Upon encountering novel widgets, their ability to generalize these actions determines their ability to perceive affordances. We implement this theory in a virtual robot model, which demonstrates human-like adaptation of affordance in interactive widgets tasks. While its predictions align with trends in human data, humans are able to adapt affordances faster, suggesting the existence of additional mechanisms.

How do people interact with computers? This fundamental question was asked by Card, Moran, and Newell in 1983 with a proposition to frame it as a question about human cognition – in other words, as a matter of how information is processed in the mind. Recently, the question has been reframed as one of adaptation: how do people adapt their interaction to the limits imposed by cognition, device design, and environment? The paper synthesizes advances toward an answer within the theoretical framework of computational rationality. The core assumption is that users act in accordance with what is best for them, given the limits imposed by their cognitive architecture and their experience of the task environment. This theory can be expressed in computational models that explain and predict interaction. The paper reviews the theoretical commitments and emerging applications in HCI, and it concludes by outlining a research agenda for future work.

Forward biomechanical simulation in HCI holds great promise as a tool for evaluation, design, and engineering of user interfaces. Although reinforcement learning (RL) has been used to simulate biomechanics in interaction, prior work has relied on unrealistic assumptions about the control problem involved, which limits the plausibility of emerging policies. These assumptions include direct torque actuation as opposed to muscle-based control; direct, privileged access to the external environment, instead of imperfect sensory observations; and lack of interaction with physical input devices. In this paper, we present a new approach for learning muscle-actuated control policies based on perceptual feedback in interaction tasks with physical input devices. This allows modelling of more realistic interaction tasks with cognitively plausible visuomotor control. We show that our simulated user model successfully learns a variety of tasks representing different interaction methods, and that the model exhibits characteristic movement regularities observed in studies of pointing. We provide an open-source implementation which can be extended with further biomechanical models, perception models, and interactive environments.

In this paper, we propose a normative approach to modeling apparently human irrational decision making (cognitive biases) that makes use of inherently rational computational mechanisms. We view preferential choice tasks as sequential decision making problems and formulate them as Partially Observable Markov Decision Processes (POMDPs). The resulting sequential decision model learns what information to gather about which options, whether to calculate option values or make comparisons between options and when to make a choice. We apply the model to choice problems where context is known to influence human choice, an effect that has been taken as evidence that human cognition is irrational. Our results show that the new model approximates a bounded optimal cognitive policy and makes quantitative predictions that correspond well to evidence about human choice. Furthermore, the model uses context to help infer which option has a maximum expected value while taking into account computational cost and cognitive limits. In addition, it predicts when, and explains why, people stop evidence accumulation and make a decision. We argue that the model provides evidence that apparent human irrationalities are emergent consequences of processes that prefer higher value (rational) policies.

Gaze-based selection has received significant academic attention over a number of years. While advances have been made, it is possible that further progress could be made if there were a deeper understanding of the adaptive nature of the mechanisms that guide eye movement and vision. Control of eye movement typically results in a sequence of movements (saccades) and fixations followed by a ‘dwell’ at a target and a selection. To shed light on how these sequences are planned, this paper presents a computational model of the control of eye movements in gaze-based selection. We formulate the model as an optimal sequential planning problem bounded by the limits of the human visual and motor systems and use reinforcement learning to approximate optimal solutions. The model accurately replicates earlier results on the effects of target size and distance and captures a number of other aspects of performance. The model can be used to predict number of fixations and duration required to make a gaze-based selection. The future development of the model is discussed.

In this paper we report a cognitive model of how people make decisions through interaction. The model is based on the assumption that interaction for decision making is an example of a Partially Observable Markov Decision Process (POMDP) in which observations are made by limited perceptual systems that model human foveated vision and decisions are made by strategies that are adapted to the task. We illustrate the model by applying it to the task of determining whether to block a credit card given a number of variables including the location of a transaction, its amount, and the customer history. Each of these variables have a different validity and users may weight them accordingly. The model solves the POMDP by learning patterns of eye movements (strategies) adapted to different presentations of the data. We compare the model behavior to human performance on the credit card transaction task.

An important problem for HCI researchers is to estimate the parameter values of a cognitive model from behavioral data. This is a difficult problem, because of the substantial complexity and variety in human behavioral strategies. We report an investigation into a new approach using approximate Bayesian computation (ABC) to condition model parameters to data and prior knowledge. As the case study we examine menu interaction, where we have click time data only to infer a cognitive model that implements a search behaviour with parameters such as fixation duration and recall probability. Our results demonstrate that ABC (i) improves estimates of model parameter values, (ii) enables meaningful comparisons between model variants, and (iii) supports fitting models to individual users. ABC provides ample opportunities for theoretical HCI research by allowing principled inference of model parameter values and their uncertainty.

While it is well known that menu usage follows a Zipfian distribution, there has been little interest in the impact of menu item frequency distribution on user's behavior. In this note, we explore the effects of frequency distribution on average menu performance as well as individual item performance. We compare three frequency distributions of menu item usage: Uniform; Zipfian with s=1 and Zipfian with s=2. The results show that (1) user's behavior is sensitive to different frequency distributions at both menu and item level; (2) individual item selection time depends on, not only its frequency, but also the frequency of other items in the menu. Finally, we discuss how these findings might have impacts on menu design, empirical studies and menu modeling.

One reason that human interaction with technology is difficult to understand is because the way in which people perform interactive tasks is highly adaptive. One such interactive task is menu search. In the current article we test the hypothesis that menu search is rationally adapted to (1) the ecological structure of interaction, (2) cognitive and perceptual limits, and (3) the goal to maximise the trade-off between speed and accuracy. Unlike in previous models, no assumptions are made about the strategies available to or adopted by users, rather the menu search problem is specified as a reinforcement learning problem and behaviour emerges by finding the optimal Markov Decision Process (MDP). The model is tested against existing empirical findings concerning the effect of menu organisation and menu length. The model predicts the effect of these variables on task completion time and eye movements. The discussion considers the pros and cons of the modelling approach relative to other well-known mod- elling approaches.

This paper presents a novel mathematical model for visual search and selection time in linear menus. Assuming two visual search strategies, serial and directed, and a pointing sub-task, it captures the change of performance with five fac- tors: 1) menu length, 2) menu organization, 3) target position, 4) absence/presence of target, and 5) practice. The novel aspect is that the model is expressed as probability density distribution of gaze, which allows for deriving total selection time. We present novel data that replicates and extends the Nielsen menu selection paradigm and uses eye-tracking and mouse tracking to confirm model predictions. The same parametrization yielded a high fit to both menu selection time and gaze distributions. The model has the potential to improve menu designs by helping designers identify more effective solutions without conducting empirical studies.

In this paper we investigate how people use online rating information to inform decision making. We examine whether a theory of searching for information to discriminate between alternative choices can explain behavior, and we contrast it to the normative theory. Partly in accord with the theory, findings from a controlled experiment suggest that in an environment dominated by positive reviews, such as the World-Wide Web, people gather more information for the best alternative under consideration, and they take more time to inspect reviews of lower rating. We discuss the theoretical and experimental implications, and propose a bounded optimal account of the way in which people acquire information in service of decision making.

We propose that a fundamental property of human psychology, the need to maintain independent social spheres, imposes constraints on the use of social network sites (SNS). We particularly focus on the consequences of visibility of communications across social spheres, and we hypothesize that technological features of SNS may bring social spheres in conflict, thus leading to increased levels of online social tension. A survey study among Facebook users was conducted to test this hypothesis. Results showed that diversity of the Facebook network predicted online tension. Moreover, the number of kin in a Facebook network was a crucial component because it predicted online tension whereas number of work and social contacts did not. Further, evidence was found to support the idea that tension might impose an upper limit on network size. We conclude with a discussion of these findings and describe how they support the thrust of recent modifications to SNS designs.

We investigate how people adapt their strategy for interleaving multiple concurrent tasks to varying objectives. A study was conducted in which participants drove a simulated vehicle and occasionally dialed a telephone number on a mobile phone. Experimental instructions and feedback encouraged participants to focus on either driving or dialing. Results show that participants adapted their task interleaving strategies to meet the required task objective, but in a manner that was nonetheless intricately shaped by internal psychological constraints. In particular, participants tended to steer in between dialing chunks of digits even when extreme vehicle drift implied that more reactive strategies would have generated better lane keeping. To better understand why drivers interleaved tasks at chunk boundaries, a modeling analysis was conducted to derive performance predictions for a range of dialing strategies. The analysis supported the idea that interleaving at chunk boundaries efficiently traded the time given up to dialing with the maintenance of a central lane position. We discuss the implications of this work in terms of contributions to understanding how cognitive constraints shape strategy adaptations in dynamic multitask environments.

An important question for HCI is to understand how and why visual search strategy is adapted to the demands imposed by the task of searching the results of a search engine. There is emerging evidence that a key part of the answer concerns the expected information gain of each of the set of available information gathering actions. We build on previous research to show that people are acutely sensitive to differences in the spacing and in the number of items returned by the search engine. These factors cause shifts in the efficiency of the available information gathering actions. We focus on an image browsing task, and show that, as a consequence of changes to the efficiency of available actions, people make small but significant changes to eye-movement strategy.

When searching a novel Web page, people often estimate the likelihood that labeled links on the page will lead to their goal. A rational analysis of this activity suggests that people should adjust their estimate of the likelihood that any one item will lead to the goal in a manner that is sensitive to the context provided by the likelihoods that other items on the page will lead to the goal. Two experiments were designed to provide evidence to discriminate between this account and others found in the literature (e.g., satisficing and assess-all accounts). The experiments systematically manipulated the relevance of the distractor items and the location of the target item on the page. The results showed that (a) a high-value item was more likely to be selected when it was first encountered if the relevance of competing distractors was relatively low and (b) more items were assessed prior to selection when the distractors were of greater semantic relevance to the goal. The location manipulation showed that if more distractors were assessed prior to the target item, then the relevance of the distractors had a greater influence on the decision as to whether to select the target immediately. These results suggest that decisions as to when to select an item from the page are sensitive to the context provided by the likelihoods of all of the items so far assessed and not just to the most recent item. The findings are therefore inconsistent with both satisficing and assess-all accounts of interactive search.

The paper describes an approach to modeling the strategic variations in performing secondary tasks while driving. In contrast to previous efforts that are based on simulation of a cognitive architecture interacting with a task environment, we take an approach that develops a cognitive constraint model of the interaction between the driver and the task environment in order to make inferences about dual-task performance. Analyses of driving performance data reveal that a set of simple equations can be used to accurately model changes in the lateral position of the vehicle within the lane. The model quantifies how the vehicle's deviation from lane center increases during periods of inattention, and how the vehicle returns to lane center during periods of active steering. We demonstrate the benefits of the approach by modeling the dialing of a cellular phone while driving, where drivers balance the speed in performing the dial task with accuracy (or safety) in keeping the vehicle centered in the roadway. In particular, we show how understanding, rather than simulating, the constraints imposed by the task environment can help to explain the costs and benefits of a range of strategies for interleaving dialing and steering. We show how particular strategies are sensitive to a combination of internal constraints (including switch costs) and the trade-off between the amount of time allocated to secondary task and the risk of extreme lane deviation.

It has been well established in Cognitive Psychology that humans are able to strategically adapt performance, even highly skilled performance, to meet explicit task goals such as being accurate (rather than fast). This paper describes a new capability for generating multiple human performance predictions from a single task specification as a function of different performance objective functions. As a demonstration of this capability, the Cognitive Constraint Modeling approach was used to develop models for several tasks across two interfaces from the aviation domain. Performance objectives are explicitly declared as part of the model, and the CORE (Constraint-based Optimal Reasoning Engine) architecture itself formally derives the detailed strategies that are maximally adapted to these objectives. The models are analyzed for emergent strategic variation, comparing those optimized for task time with those optimized for working memory load. The approach has potential application in user interface and procedure design.

This paper presents X-PRT, a new cognitive modeling tool supporting activities ranging from interface design to basic cognitive research. X-PRT provides a graphical model development environment for the CORE constraint-based cognitive modeling engine [7,13,21]. X-PRT comprises a novel feature set: (a) it supports the automatic generation of predictive models at multiple skill levels from a single task-specification, (b) it supports a comprehensive set of modeling activities, and (c) it supports compositional reuse of existing cognitive/perceptual/motor skills by transforming high-level, hierarchical task descriptions into detailed performance predictions. Task hierarchies play a central role in X-PRT, serving as the organizing construct for task knowledge, the locus for compositionality, and the cognitive structures over which the learning theory is predicated. Empirical evidence supports the role of task hierarchies in routine skill acquisition.

In this paper we report a new approach to generating predictions about skilled interactive cognition. The approach, which we call Cognitive Constraint Modeling, takes as input a description of the constraints on a task environment, on user strategies, and on the human cognitive architecture and generates as output a prediction of the time course of interaction. In the Cognitive Constraint Models that we have built this is achieved by encoding the assumptions inherent in CPM-GOMS as a set of constraints and reasoning about them using finite domain constraint satisfaction.

Two important phenomena in human - computer interaction (HCI) are considered: the reliance on external information rather than memory, and the interleaving of planning and action. These phenomena are important, it is argued, because they challenge some particular cognitive models. However, we reject those views, influential in the HCI literature, that phenomena like these require radically new conceptions of cognition or behaviour. It is shown that the phenomena are not universal laws of behaviour, but that instead people decide how much to remember and how much to plan according to a consideration of the costs and benefits of different strategies. Thus the classical cognitive conception of humans as adaptive decision makers is vital for a deep understanding of HCI.

What is the role of a cognitive architecture in shaping a model built within it? Compared with a model written in a programming language, the cognitive architecture offers theoretical constraints. These constraints can be "soft," in that some ways of constructing a model are facilitated and others made more difficult, or they can be "hard," in that certain aspects of a model are enforced and others ruled out. We illustrate a variety of these possibilities. In the case of Soar, its learning mechanism is sufficiently constraining that it imposes hard constraints on models constructed within it. We describe how one of these hard constraints deriving from Soar's learning mechanism ensures that models constructed within Soar must learn a display-based skill and, other things being equal, must find display-based devices easier to learn than keyboard-based devices. We discuss the relation between architecture and model in terms of the degree to which a model is "compliant" with the constraints set by the architecture. Although doubts are sometimes expressed as to whether cognitive architectures have any empirical consequences for user modeling, our analysis shows that they do. Architectures play their part by imposing theoretical constraints on the models constructed within them, and the extent to which the influence of the architecture shows through in the model's behavior depends on the compliancy of the model.

This article advances a heuristic for conceptual instruction, based on the yoked‐state space (YSS) hypothesis about the contents of users’ device models (Payne, 1987a; Payne, Squibb, & Howes, 1990). The YSS hypothesis suggests that users of a system need to understand the system's representation of the task domain. Accordingly, conceptual instructions should express the mapping from device objects onto goal‐space objects, especially those aspects that are not visible at the user interface. Such instructions are developed for a menu‐driven computer system based on the RATES system for performing remote diagnosis of telephone lines. A simple comparative experiment shows that novices who receive these instructions learn to use RATES more quickly than novices who receive only background instruction and a brief procedural manual. These results increase empirical support for the YSS hypothesis, and, at the same time, suggest a heuristic for the design of conceptual instructions.

Programmable User Models (PUMS) are tools based on psychological theory that enable interface designers to predict the usability of a proposed design.This paper presents a variant in which the PUM, implemented in Soar and incorporating the constraints of Display-based Task-Action Grammars, learns the task-action mapping by being guided by the designer during performance.We show that the more consistent and interactive the interface, the easier it is for the designer to teach the PUM the necessary taskaction mapping. PREDICTING INTERFACE LEARNABILITYA Programmable User Model (PUM, [14]) is a psychologically constrained architecture in which the interface designer is invited to program a simulation of a user.By making the user's knowledge explicit, PUMS

This paper addresses the problem of how a novice computer user, engaged in exploratory learning, accounts for the behaviour of the device. Exploratory learning is the norm for many users who encounter computers in the work place. Exploratory learners must acquire methods from a suboptimal stream of task directed behaviour and its observable effects.

To construct a conceptual model of a device, the user must conceptualize the device's representation of the task domain. This knowledge can be represented by three components: a device-based problem space, which specifies the ontology of the device in terms of the objects that can be manipulated and their interrelations, plus the operators that perform the manipulations; a goal space, which represents the objects in terms of which user's goals are expressed; and a semantic mapping, which determines how goal space objects are represented in the device space. The yoked state space (YSS) model allows an important distinction concerning the mental representation of procedures. If a step in a procedure specifies a transformation of the user's device space, then it has an autonomous meaning for the user, independent of its role in the sequence or method. The device space provides a figurative account of the operator. However, some operators do not affect the minimal device space, and their only meaning for the user derives from their role in a method: The method affords an operational account of the operator. Figurative accounts can be constructed from operational accounts only by elaborating the device space with new concepts. The YSS is illustrated through a simple description of a device model for a cut-and-paste text editor. Three experiments addressed the claims of this model. The first experiment used a sorting paradigm to show that users do acquire the novel device space concept of a string of adjacent characters (including space and return). The second and third experiments asked novices to make inferences about text editor behavior on the basis of simple demonstrations. They showed that (a) the availability of the string concept is critically dependent on the details of interface design, (b) figurative accounts of the copy operation afford more efficient methods and may be promoted by appropriate names for procedure steps, and (c) a conceptual model may transfer from one device to another. Together, the three experiments supported the YSS hypothesis.