• Both gaze and paragaze can indicate task transitions due to interruption or not • Both gaze and paragaze can reflect task change due to interruption • Only strong emotional interruption affects eye behaviors afterwards • Task type significantly affects the time to attend an interruption • Unproportionally increased completion time for low and high cognitive load tasks One technology adoption outcome is ever more frequent task interruptions, which makes interruption management a central human computer interface design problem. Studies on interruption often focused on the link between interruptions and genesis of error and eye gaze to validate associated task cues engagement. How eye behaviors reflect load type and level changes between primary and interrupting tasks and how emotional interruptions impact mental state in primary tasks – factors that contribute to cognitive processing – remain unanswered. In this study, we analyzed performance metrics and eye behaviors from 18 participants while they completed an uninterrupted and interrupted roleplay task. Our results reveal that although interruption notification could affect the time to attend to an interruption, high cognitive load resulted in a detrimental impact on completion time and low cognitive load led to time reset for primary tasks. No existing evidence suggests that affect type influences the time metrics, however, high arousal and valence induced by interruptions altered eye behavior upon returning to the primary task, indicating less visual information taken and under a high load level, which showed evidence of the impact of affective interruption for the first time. Noteworthily, the eye behaved differently at interruption beginning compared with the end and uninterrupted task transitions, indicating the feasibility of including eye behavior for interruption estimation. Overall, these results are the first experimental evidence for theories that interruption can cause high mental load, posit extra mental effort, forgetting primary tasks and behaviors being directed by the current most active goal.

Persuasion is to change or influence a person's attitude or behavior without coercion or deception. Combined with personalization, personalized persuasion has demonstrated greater impact on influence. From persuasion theories in social psychology, emotion, cognition and personality are considered important factors. These factors are also under active research for automated detection within the computing community. How mental state and personality play a role in the process of persuasion and in persuasion outcomes has not been investigated. In this paper, we investigated the difference of mental state and personality between persuaders and receivers in a dyadic conversation where different persuasion outcome was produced. A video dataset of 24 participants in pairs debating in a survival task was collected. Arousal, valence, gaze direction, speech length and Big Five Inventory were annotated and processed to represent emotion, attention, mental load and personality during each dialog. Statistical analysis results show that persuaders’ arousal was significantly higher than receivers’ to disagreement outcome, persuaders imposed lower mental load on receivers to agreement outcome, and when persuaders’ extraversion score was higher than receivers’, the agreement response rate was higher. This research contributes to the understanding of mental state and personality of both persuaders and receivers in the context of dyadic interaction for a potential of dynamic and personalized adaptation.

Mental state induced by different task contexts and load levels is of interest for human health and wellness, and eye activity extracted from infrared eye images is well-suited to estimate it. As a useful tool for emotion recognition, facial action units (FAUs) extracted from facial images are well-established, however these are insufficiently detailed for the eye. In this paper, we extract discrete eye behaviors from eyelid, iris and pupil boundaries and propose eye action units (EAUs) based on the eye appearance (behavior), providing a detailed and interpretable representation that shares the advantages of FAUs and describes the wide range of eye states and shapes. Eight volunteers annotated 11 EAUs for 120 eye images, represented by a series of discrete eye behaviors. Analysis shows that the EAUs can be viably characterized by fundamental eye behaviors with moderate to substantial agreement. When evaluating discrete eye behaviors and EAUs for recognition of four mental states and two load levels, the former achieved significantly higher accuracy than conventional features of pupil size change and blink rate, especially using behavior duration features, and EAUs outperformed combinations of discrete eye behaviors in general, implying their utility as an action unit.

The International Society for Clinical Eye Tracking (ISCET) serves as a global platform for promoting international consensus on open standards on clinical eye tracking. Originally formed in March 2023, ISCET was created to facilitate collaboration, knowledge exchange, and advancements in clinical eye tracking applications, with the ultimate goal of fostering interdisciplinary research and improving clinical outcomes. Through collaborative and interdisciplinary efforts, ISCET’s current mission is to provide guidance on conducting eye-tracking tasks in clinical settings, unify clinicians’ voices, and maintain reference datasets for normative comparisons. ISCET now has 80 members, spread across the globe. In the Europe/Africa region meeting on January 24, 2024, ISCET established subcommittees to address specific needs. This paper outlines the rationale behind ISCET’s formation, its mission, objectives, ongoing initiatives, and organizational structure. Ongoing work focuses on surveying current international clinical eye tracker usage to inform standards development.

The International Society for Clinical Eye Tracking (ISCET) serves as a global platform for promoting international consensus on open standards in the application of eye tracking to clinical areas of focus. Formed in March 2023, ISCET facilitates collaboration, knowledge exchange, and advancements in clinical eye-tracking applications, with the goal of fostering interdisciplinary research and improving clinical outcomes. Through collaborative and interdisciplinary efforts, ISCET aims to provide guidance on conducting eye-tracking tasks in clinical settings, unify clinicians’ voices, and maintain reference datasets for normative comparisons. Convening during its latest meeting for the Europe/Africa region on January 24, 2024, ISCET established subcommittees to address specific needs. This paper outlines the rationale behind ISCET’s formation, its mission, objectives, ongoing initiatives, and organizational structure. Ongoing work focuses on surveying current international clinical eye tracker usage to inform standards development.

During a naturalistic conversation, we can easily perceive whether a person is speaking or listening and when they are about to start or finish speaking, which we refer to as conversational states in this paper. Building this ability into a machine can be helpful in a wide variety of human-machine collaboration contexts for effective communication. A wealth of evidence from psychology and neuroscience suggests that the eye is a reliable window into human internal states since it senses the perceived changes in the outside world. In this study, we investigate the relationship between eye behavior and four conversational states for the first time and examine the viability of automatically recognizing the conversational states. The results demonstrate that eye center relative to the head is not a good indicator to distinguish conversational states, but the distribution, frequency and duration of eye states are strongly correlated with conversational states while eyelid is associated with conversational state in certain extent. The accuracy of recognition of the four conversational states using the proposed eye behaviors is well above chance level, and outperforms baselines using pupillary response, pupil center position, and blink rate. This finding suggests that eye state and eyelid shape contains valuable information about conversational states but have been overlooked in previous studies. It is promising to include eye behavior in wearable interfaces for dialogue systems and for social signal processing, with the further advantages of being ‘always on’, less sensitive to luminance variability, and better privacy preserving compared with using facial image and speech data.

Sensing the mental state induced by different task contexts, where cognition is a focus, is as important as sensing the affective state where emotion is induced in the foreground of consciousness, because completing tasks is part of every waking moment of life. However, few datasets are publicly available to advance mental state analysis, especially those using the eye as the sensing modality with detailed ground truth for eye behaviors. In this study, we contribute a high-quality publicly accessible eye video dataset, IREye4Task, where the eyelid, pupil and iris boundary are annotated for each frame to obtain eye behaviors as responses to four different task contexts and two load levels of tasks, over more than a million frames. Meanwhile, we propose a series of eye behavior representations to provide insights into how the eye behaves during different mental states. Finally, we benchmark three mental-state recognition tasks for this dataset to demonstrate the effectiveness of the eye behavior representations. This is the first public wearable eye video dataset for mental state analysis with high quality eye landmarks and a variety of mental states, and is the first study analyzing comprehensive eye behaviors far beyond using pupil size and blink in previous studies.

Eye activity has previously been found to be relevant to emotion, cognitive load and task transition, but studies have mainly focused on one of them and used a single type of task. This motivates an investigation of eye activity for emotion, cognitive load and task transition recognition in a single study with the goal of understanding the capability of eye activity. We recorded 15 participants’ eye data while they completed a sequence of free-viewing emotional image tasks and a sequence of arithmetic tasks. Eye activity features within fixed analysis windows were extracted to classify (i) between four levels of arousal, (ii) three levels of valence, (iii) between three levels of cognitive load, (iv) between affective and cognitive tasks, and (v) between task transition and non-transition. The results suggest that eye activity can best be used for task transition recognition, with an accuracy of 77%, followed by cognitive load level recognition, then affective and cognitive task recognition. Implications of this finding include automatically segmenting long and continuous signals into task units with eye activity before analyzing human behavior.

Automatic detection of fatigue based on remote cameras and computer vision has been investigated for many years, and is a viable solution in many contexts, but not for mobile contexts. This paper investigates eye activity extraction methods for wearable fatigue detection using low-cost hardware that is becoming ubiquitous in glasses form-factors, and proposes a new measure for fatigue based on the synchronization between blink and saccade. Evaluation on a novel dataset shows that the novel blink-saccade synchronization measure achieves statistically significant separation of control and fatigued participants, and provides automatic fatigue detection accuracy improvements of 10% relative to existing saccade measures based on velocity and amplitude.

Eye movement plays an important role in cognition and perception, and the detection of saccade and fixation has been studied for human-computer applications, however often under conditions where head movement is constrained, and often using calibration-dependent gaze information rather than the raw pupil position. In order to investigate the performance of saccade and fixation detection using gaze and pupil data, three representative saccade detection algorithms are applied to both pupil data and gaze data collected with and without head movement, and their performance is evaluated against a stimulus-induced ground truth under different measures. Results indicate that saccade/fixation detection using pupil data generally provides better performance than using gaze data with an 8.6% improvement in Cohen’s Kappa (averaged across the three algorithms), even when moderate head movement is involved. Hence, pupil data can be used as an alternative to gaze data for saccade and fixation detection in wearable contexts with less effort in calibration and higher accuracy.

For longitudinal behavior analysis, task type is an inevitable and important variable. In this article, we propose an event-based behavior modeling approach and employ non-invasive wearable sensing modalities (eye activity, speech and head movement) to recognize task load level under four different task load types. The novelty lies in converting physiological and behavioral signals into meaningful events and utilizing their sequence across multiple modalities to distinguish load levels and types. We evaluated this approach on head-worn sensor data from 24 participants completing four different tasks for recognizing (i) low and high load level for a given task load type, (ii) low and high load level regardless of load type, and (iii) both load level and load type. Findings show that the recognition rate is reasonable in (i), close to chance level in (ii), and well above chance level in (iii) for 8 classes using participant-dependent and -independent schemes. Further, a fusion of the proposed event-based features and conventional continuous features achieved the best or similar performance in most cases. These results suggest that task type needs to be considered when using continuous features and that the proposed event-based modeling paradigm is promising for longitudinal behavior analysis.

Humans always engage multiple modalities when performing tasks, such as eye activity, speech and head movement, which contain rich information indicative of task load that can help understand and predict human psychological state and behavior. In recent research into multimodal signal processing, the ideas of sequence- and coordination-based event features have been proposed, which explicitly utilize the interaction information among different modalities. In this paper, we propose event intensity and event duration-based features, which capture the extent and duration of onset events that denote major changes in behavior signal. These features are combined with sequence- and coordination-based event features to achieve state-of-the-art performance in assessing task load levels and load types. In experimental work, we collected eye activity, speech and head movement data from 24 participants during cognitive, perceptual, physical and communication tasks. Results suggest that by fusing these four compact, interpretable event-based features, strong accuracy can be achieved: 84% for two load level classification, 89% for four load type classification and 76% for 8-class classification, outperforming conventional statistical features and deep neural network self-learned features by up to 9% and 25% respectively. These features do not need to be selected during training and can generalize well for different participants and different task types.

Physiological and behavioral measures allow computing devices to augment user interaction experience by understanding their mental load. Current techniques often utilize complementary information between different modalities to index load level typically within a specific task. In this study, we propose a new approach utilizing the timing between physiology/behavior change events to index low and high load level of four task types. Findings from a user study where eye, speech, and head movement data were collected from 24 participants demonstrate that the proposed measures are significantly different between low and high load levels with high effect size. It was also found that voluntary actions are more likely to be coordinated during tasks. Implications for the design of multimodal-multisensor interfaces include (i) utilizing event change and interaction in multiple modalities is feasible to distinguish task load levels and load types and (ii) voluntary actions should be allowed for effective task completion.

Physical activity recognition using wearable sensors has achieved good performance in discriminating heterogeneous activities for health monitoring, but there has been less investigation of sedentary activities, e.g., desk work, which is often physically homogenous, to improve health in office environments. In this study, we explored head movement as a new sensing modality for physical and mental activity analysis. A new algorithm which segments gyroscope signals into atomic head movement events is proposed. Instead of recognizing activities in terms of predefined categories, we recognized four dimensions of task load: cognitive, perceptual, communicative, and physical, analogous to current manual workload assessment methods like NASA-TLX. We collected head movement data from 24 participants who wore a tri-axial inertial sensor at head while performing multiple tasks with varying load levels at office. An average of 70% accuracy was achieved for recognizing cognitive load levels, and more than 80% for the other three load types. The proposed event features outperformed a set of 181 features from previous physical activity recognition studies. We also demonstrated that these atomic event features are diagnostic of different load types in cross-load type classification, showing the promise of physical and mental load monitoring for health.

Minimizing user cognitive load is suggested as an integral part of human-centered design, where a more intuitive, easy to learn, and adaptive interface is desired. In this context, it is difficult to develop optimal strategies to improve the design without first knowing how user cognitive load fluctuates during interaction. In this study, we investigate how cognitive load measurement is affected by different task types from the perspective of the load theory of attention, using pupil diameter and blink measures. We induced five levels of cognitive load during low and high perceptual load tasks and found that although pupil diameter showed significant effects on cognitive load when the perceptual load was low, neither blink rate nor pupil diameter showed significant effects on cognitive load when the perceptual load was high. The results indicate that pupil diameter can index cognitive load only in the situation of low perceptual load and are the first to provide empirical support for the cognitive control aspect of the load theory of attention, in the context of cognitive load measurement. Meanwhile, blink is a better indicator of perceptual load than cognitive load. This study also implies that perceptual load should be considered in cognitive load measurement using pupil diameter and blink measures. Automatic detection of the type and level of load in this manner helps pave the way for better reasoning about user internal processes for human-centered interface design.

Monitoring pupil and blink dynamics has applications in cognitive load measurement during human-machine interaction. However, accurate, efficient, and robust pupil size and blink estimation pose significant challenges to the efficacy of real-time applications due to the variability of eye images, hence to date, require manual intervention for fine tuning of parameters. In this paper, a novel self-tuning threshold method, which is applicable to any infrared-illuminated eye images without a tuning parameter, is proposed for segmenting the pupil from the background images recorded by a low cost webcam placed near the eye. A convex hull and a dual-ellipse fitting method are also proposed to select pupil boundary points and to detect the eyelid occlusion state. Experimental results on a realistic video dataset show that the measurement accuracy using the proposed methods is higher than that of widely used manually tuned parameter methods or fixed parameter methods. Importantly, it demonstrates convenience and robustness for an accurate and fast estimate of eye activity in the presence of variations due to different users, task types, load, and environments. Cognitive load measurement in human-machine interaction can benefit from this computationally efficient implementation without requiring a threshold calibration beforehand. Thus, one can envisage a mini IR camera embedded in a lightweight glasses frame, like Google Glass, for convenient applications of real-time adaptive aiding and task management in the future.

A day in the life of a user can be segmented into a series of tasks: a user begins a task, becomes loaded perceptually and cognitively to some extent by the objects and mental challenge that comprise that task, then at some point switches or is distracted to a new task, and so on. Understanding the contextual task characteristics and user behavior in interaction can benefit the development of intelligent systems to aid user task management. Applications that aid the user in one way or another have proliferated as computing devices become more and more of a constant companion. However, direct and continuous observations of individual tasks in a naturalistic context and subsequent task analysis, for example the diary method, have traditionally been a manual process. We propose a method for automatic task analysis system, which monitors the user's current task and analyzes it in terms of the task transition, and perceptual and cognitive load imposed by the task. An experiment was conducted in which participants were required to work continuously on groups of three sequential tasks of different types. Three classes of eye activity, namely pupillary response, blink and eye movement, were analyzed to detect the task transition and non-transition states, and to estimate three levels of perceptual load and three levels of cognitive load every second to infer task characteristics. This paper reports statistically significant classification accuracies in all cases and demonstrates the feasibility of this approach for task monitoring and analysis.

The measurement of a user's mental effort is a problem whose solutions may have important applications to adaptive interfaces and interface evaluation. Previous studies have empirically shown links between eye activity and mental effort; however these have usually investigated only one class of eye activity on tasks atypical of HCI. This paper reports on research into eight eye activity based features, spanning eye blink, pupillary response and eye movement information, for real time mental effort measurement. Results from an experiment conducted using a computer-based training system show that the three classes of eye features are capable of discriminating different cognitive load levels. Correlation analysis between various pairs of features suggests that significant improvements in discriminating different effort levels can be made by combining multiple features. This shows an initial step towards a real-time cognitive load measurement system in human-computer interaction.