In affect-aware task adaptation, users’ psychological states are recognized with diverse measurements and used to adapt computer-based tasks. User experience with such adaptation improves as the accuracy of psychological state recognition and task adaptation increases. However, it is unclear how user experience is influenced by algorithmic transparency: the degree to which users understand the computer's decision-making process. We thus created an affect-aware task adaptation system with 4 algorithmic transparency levels (none/low/medium/high) and conducted a study where 93 participants first experienced adaptation with no transparency for 16 minutes, then with one of the other 3 levels for 16 minutes. User experience questionnaires and physiological measurements (respiration, skin conductance, heart rate) were analyzed with mixed 2×3 analyses of variance (time × transparency group). Self-reported interest/enjoyment and competence were lower with low transparency than with medium/high transparency, but did not differ between medium and high transparency. The transparency level may also influence participants’ respiratory responses to adaptation errors, but this finding is based on ad-hoct-tests and should be considered preliminary. Overall, results show that the degree of algorithmic transparency does influence self-reported user experience. Since transparency information is relatively easy to provide, it may represent a worthwhile design element in affective computing.

Physiological sensors are commonly applied for user state monitoring and consequent machine behavior adaptation in applications such as rehabilitation and intelligent cars. While more accurate user state monitoring is known to lead to better user experience, increased accuracy often requires more sensors or more complex sensors. The increased setup time and discomfort involved in the use of such sensors may itself worsen user experience. To examine this effect, we conducted a study where 72 participants interacted with a computer-based multitasking scenario whose difficulty was periodically adapted - ostensibly based on data from either a remote eye tracker or a lab-grade “wet” electroencephalography sensor. Deception was used to ensure consistent difficulty adaptation accuracies, and user experience was measured with the Intrinsic Motivation Inventory, NASA Task Load Index, and an ad-hoc scale. We found few user experience differences between the eye tracker and electroencephalography sensor - while one interaction effect was noted, it was small, and there were no other differences. This result is at first surprising and seems to indicate that comfort and setup time are not major factors for laboratory-based user experience evaluations of such technologies. However, the result is likely due to a suboptimal study protocol where each participant interacted with only one sensor. In future work, we will use an alternate protocol to further explore the effects of user comfort and setup time on user experience.

Two people's physiological responses become more similar as those people talk or cooperate, a phenomenon called physiological synchrony. The degree of synchrony correlates with conversation engagement and cooperation quality, and could thus be used to characterize interpersonal interaction. In this study, we used a combination of physiological synchrony metrics and pattern recognition algorithms to automatically classify four different dyadic conversation scenarios: two-sided positive conversation, two-sided negative conversation, and two one-sided scenarios. Heart rate, skin conductance, respiration and peripheral skin temperature were measured from 16 dyads in all four scenarios, and individual as well as synchrony features were extracted from them. A two-stage classifier based on stepwise feature selection and linear discriminant analysis achieved a four-class classification accuracy of 75.0% in leave-dyad-out crossvalidation. Removing synchrony features reduced accuracy to 65.6%, indicating that synchrony is informative. In the future, such classification algorithms may be used to, e.g., provide real-time feedback about conversation mood to participants, with applications in areas such as mental health counseling and education. The approach may also generalize to group scenarios and adjacent areas such as cooperation and competition.

Classification algorithms are used in affective computing to classify the state of the user and adapt the computer's behaviour, but it is unclear how classification accuracy influences the overall user experience. We present a study in which classification accuracy is artificially pre-defined and used to adapt to the difficulty of a video game. Eighty subjects played the game and were told that difficulty would be adapted according to the measured brain activity. They played the game twice, with different classification accuracies, and then reported different aspects of their overall game experience using questionnaires. Classification accuracy was correlated with both in-game fun (r = 0.46) and satisfaction with the difficulty adaptation (r = 0.56). Most subjects could perceive a difference between two classification accuracies that differed by 16.7%. We tentatively posit that, for affective video games, an acceptable classification accuracy is 70–80%. Furthermore, studies that attempt to improve affect classification accuracy should aim for a practically meaningful improvement of 10%.

The papers in this special section focus on commercial applications for affective computing. One of the main goals of affective computing is to create machines which can adapt to users’ emotions in order to produce more natural and efficient interaction. Emotion recognition is thus a central component of the field, and is based on a variety of measurements (facial expressions, speech, gait patterns, physiology, eye tracking, etc.) that are analyzed using advanced pattern recognition techniques. Furthermore, researchers and entrepreneurs have identified countless possible applications of affect-aware technology, from health and driver monitoring to exercise [6] and computer game adaptation. However, although great scientific advances have been made and many applications have been proposed, only few robust implementations have been presented or validated, and commercial adoption of affect-aware technology has been marginal. One rare example of a successful application is the smile detector used in digital cameras to automatically take pictures when the subject is smiling. This weak adoption of the technology can be attributed to several unsolved challenges in the domain of affective computing.

This paper uses physiological measurements to estimate human workload and effort in physical human–robot interaction. Ten subjects performed 19 consecutive task periods using the ARMin robot while difficulty was varied along two scales. Three physiological modalities were measured: electroencephalography, autonomic nervous system (ANS) responses (electrocardiography, skin conductance, respiration, skin temperature) and eye tracking. After each task period, reference workload and effort values were collected using the NASA Task Load Index. Machine learning was used to estimate workload and effort from physiological data. All three physiological modalities performed significantly better than random, particularly using nonlinear estimation algorithms. The most important ANS responses were respiration and skin conductance, while the most important electroencephalographic information was obtained from frontal and central sites. However, all three physiological modalities were outperformed by task performance and movement data. This suggests that future studies should try to demonstrate advantages of physiological measurements over other information sources.

Rehabilitation robots have the potential to greatly improve motor rehabilitation. However, the patient must be properly motivated to actively participate in therapy. Several strategies have been suggested to improve patient motivation, but one element has not yet been explored: playing with other people. We designed a two-player rehabilitation game played by two people using two ARMin III robots. We tested three game modes: single-player (competing against a computer), competitive (competing against a human), and cooperative (cooperating with a human against a computer). All modes were played by 24 healthy subjects who filled out questionnaires about their personality and in-game motivation. Almost all subjects preferred playing the two-player game modes to the single-player one, as they enjoyed being able to talk and interact with another person. However, there were two distinct player groups. One group liked the competitive mode but not the cooperative mode while the other liked the cooperative but not the competitive mode. Subjects who liked the competitive mode also put more effort into it. Finally, subjects' personalities partially predicted what mode they would like. This emphasizes that two-player rehabilitation games have advantages over single-player ones, but that the right game needs to be chosen for each subject. An extended patient study is planned for the near future.

Functional near-infrared spectroscopy (fNIRS) is a noninvasive optical method that measures cortical activity based on hemodynamics in the brain. Physiological signals (biosignals), such as blood pressure and respiration, are known to appear in cortical fNIRS recordings. Some biosignal components occupy the same frequency band as the cortical response, and respond to the subjects activity. To process an fNIRS signal in a brain-computer interface, it is desirable to know which components of the signal come from cortical response, and which come from biosignal interference. Numerous filtering methods have been proposed to this end with mixed success, possibly because they assume that the cortical and physiological signals combine linearly, or that biosignals do not correlate with subject behavior. Here, we propose an adaptive filter with a cost function based on mutual information to selectively remove information that correlates with blood pressure from the fNIRS signal. The filter was tested with real and simulated data. The real signals were measured on seven healthy subjects performing an isometric pinching task. Cross-correlation and mutual information were employed as performance measures. The filter successfully removed correlations between blood pressure and the fNIRS signal, by an equal or greater amount compared to a traditional recursive least squares adaptive filter. Blood pressure was found to be the most informative signal to classify rest and active periods using linear discriminant analysis. Any task information in the fNIRS signal was redundant to that expressed by blood pressure.

In an affective feedback loop, the computer maps various measurements to affective variables such as enjoyment, then adapts its behavior based on the recognized affects. The affect recognition is never perfect, and its accuracy (percentage of times the correct affective state is recognized) depends on many factors. However, it is unclear how this accuracy relates to the overall user experience. As recognition accuracy is difficult to control in a real affective feedback loop, we describe a method of simulating recognition accuracy in a game where difficulty is increased or decreased after each round. The game was played by 261 participants at different simulated recognition accuracies. Participants reported their satisfaction with the recognition algorithm as well as their overall game experience. We observed that in such a game, the affective feedback loop must adapt game difficulty with an accuracy of at least 80 percent to be accepted by users. Furthermore, users who do not enjoy the game are likely to stop playing it rather than continue playing and report low enjoyment. However, the acceptable recognition accuracy may not generalize to other contexts, and studies of affect recognition accuracies in other applications are needed.

Physiological computing represents a mode of human–computer interaction where the computer monitors, analyzes and responds to the user’s psychophysiological activity in real-time. Within the field, autonomic nervous system responses have been studied extensively since they can be measured quickly and unobtrusively. However, despite a vast body of literature available on the subject, there is still no universally accepted set of rules that would translate physiological data to psychological states. This paper surveys the work performed on data fusion and system adaptation using autonomic nervous system responses in psychophysiology and physiological computing during the last ten years. First, five prerequisites for data fusion are examined: psychological model selection, training set preparation, feature extraction, normalization and dimension reduction. Then, different methods for either classification or estimation of psychological states from the extracted features are presented and compared. Finally, implementations of system adaptation are reviewed: changing the system that the user is interacting with in response to cognitive or affective information inferred from autonomic nervous system responses. The paper is aimed primarily at psychologists and computer scientists who have already recorded autonomic nervous system responses and now need to create algorithms to determine the subject’s psychological state.

Rehabilitation robots, together with vision and audio systems form the multimodal environment for exercising the person in a number of ways, unavoidably influencing the physiological state of the subject. This paper examines viability of measuring psycho physiological responses to different robotic tasks. The heart rate, skin conductance, respiration and peripheral skin temperature were observed to verify if physical activity obstructs useful recordings and to verify responses in stroke population. 30 healthy subjects were checked with a control task, a purely mental task and task with physical load. 23 subacute stroke persons did a control task, pick and place task (+ inverted version) and Stroop test, same as 22 healthy control subjects. Psycho physiological measurements yielded results even in the presence of physical load and can thus potentially be useful for rehabilitation robotics. Similar responses as in healthy control group were found in the stroke group. Skin conductance response frequency, respiratory rate, skin conductance and skin temperature (all changes from baseline) were confirmed as parameters signaling changes in arousal and valence of both, stroke and control groups.