Learning clothes repair is challenging for novices, who face interconnected procedural and embodied challenges, especially when learning alone. Existing tools fail to provide holistic support as interactive tutors and lack awareness of the embodied interactions of working with textiles. This paper presents a multi-phase study that investigates these challenges and explores the design space for a Human-Touch-Aware conversational agent (RepairBot). We began with an in-depth autoethnography to understand the novice experience, which informed the development of the RepairBot Conversation Framework (RBCF) together with a design implementation of a technology probe. Using the RepairBot prototype together with a Wizard-of-Oz approach to simulate Human-Touch-Awareness, we investigated how a conversational agent could support repair learning in novices as well as engage them with their own clothes-repairing projects. Subsequent lab and in-home studies with novice participants suggested specific conversational and embodied mechanisms that would facilitate novices’ holistic understanding of repair, increase their confidence, and elicit attentive touch and emotional reflection. We bring these mechanisms together in the framework presented in this paper.

Advances in emerging technologies, such as on-body mechanical actuators and electrical muscle stimulation, have allowed computers to take control over our bodies. This presents opportunities as well as challenges, raising fundamental questions about agency and the role of our bodies when interacting with technology. To advance this research field as a whole, we brought together expert perspectives in a week-long seminar to articulate the grand challenges that should be tackled when it comes to the design of computers’ control over our bodies. These grand challenges span technical, design, user, and ethical aspects. By articulating these grand challenges, we aim to begin initiating a research agenda that positions bodily control not only as a technical feature but as a central, experiential, and ethical concern for future human–computer interaction endeavors.

FFAME (Filtering Familiar Audio for Movement Exploration) is a novel sonification framework aiming to facilitate movement in individuals with chronic back pain. Our personalised, music-based approach contrasts and extends prior work with predetermined tonal sonification. FFAME progressively filters selected music based on angles of the trunk. Through a qualitative analysis of reported experience of 15 participants with chronic pain and 5 physiotherapists, we identify how sonification parameters and musical characteristics affect movement and meaning-making. Music-based movement sonification proved impactful across multiple dimensions: (1) encouraging movement, (2) escaping pain-related rumination, (3) externalizing pain experiences, and (4) scaffolding physical activities. Drawing on enactivism and related philosophies, the study highlights how the semantic indeterminacy of music, combined with real-time movement sonification, created a rich, open-ended environment that supported user agency and exploration. Sonification for pain management can be creative and expressive, enabling people with pain to extend challenging movements and build movement confidence.

Pain communication varies significantly among individuals, some are highly expressive, while others demonstrate stoic restraint and offer minimal verbal indication of discomfort. Substantial progress has been made in identifying behavioral indicators of pain. A growing body of literature highlights measurable indices of pain through facial expressions, vocalizations, body movements, as well as physiological and neural responses. To enhance the reliability of pain monitoring, automated pain assessment has emerged as a promising approach. Although available datasets remain limited, they are steadily increasing, helping to drive research forward. Despite notable progress, this field is still in its early stages. The 5th edition of the AAP workshop continues to seek to highlight current research and foster interdisciplinary collaboration and discussion to accelerate progress in this important area.

Chronic pain is a prevalent condition where fear of movement and pain interfere with everyday functioning. Yet, there is no open body movement dataset for people with chronic pain in everyday settings. Our EmoPain@Home dataset addresses this with capture from 18 people with and without chronic pain in their homes, while they performed their routine activities. The data includes labels for pain, worry, and movement confidence continuously recorded for activity instances for the people with chronic pain. We explored baseline two-level pain detection based on this dataset and obtained 0.62 mean F1 score. However, extension of the dataset led to deterioration in performance confirming high variability in pain expressions for real world settings. We investigated baseline activity recognition for this setting as a first step in exploring the use of the activity label as contextual information for improving pain level classification performance. We obtained mean F1 score of 0.43 for 9 activity types, highlighting its feasibility. Further exploration, however, showed that data from healthy people cannot be easily leveraged for improving performance because worry and low confidence alter activity strategies for people with chronic pain. Our dataset and findings lay critical groundwork for automatic assessment of pain experience and behaviour in the wild.

Social media (SM) breaks from studying can either support students' wellbeing and performance by acting as a recovery behaviour or subvert it by acting as a procrastination behaviour. It is currently unclear which influences lead an SM break to be a positive recovery vs. negative procrastination behaviour. A behavioural and emotion regulation (ER) perspective may help to elucidate these influences. In this paper, we report a semi-structured interview study with 20 undergraduates to explore their experiences of SM breaks when studying. Our analysis describes how motivational and environmental factors can influence a break's propensity for recovery or procrastination during the break initiation and execution phases. We apply an ER perspective to these reports and demonstrate how it helps to explain further in which circumstances SM breaks are likely to support recovery or procrastination. Based on this analysis, we present recommendations for designing interventions to support healthy breaks and reduce unhealthy ones.

Self-supervised learning has shown value for uncovering informative movement features for human activity recognition. However, there has been minimal exploration of this approach for affect recognition where availability of large labelled datasets is particularly limited. In this paper, we propose a P-STEMR (Parallel Space-Time Encoding Movement Representation) architecture with the aim of addressing this gap and specifically leveraging the higher availability of human activity recognition datasets for pain-level classification. We evaluated and analyzed the architecture using three different datasets across four sets of experiments. We found statistically significant increase in average F1 score to 0.84 for pain level classification with two classes based on the architecture compared with the use of hand-crafted features. This suggests that it is capable of learning movement representations and transferring these from activity recognition based on data captured in lab settings to classification of pain levels with messier real-world data. We further found that the efficacy of transfer between datasets can be undermined by dissimilarities in population groups due to impairments that affect movement behaviour and in motion primitives (e.g. rotation versus flexion). Future work should investigate how the effect of these differences could be minimized so that data from healthy people can be more valuable for transfer learning.

The use of multiple raters to label datasets is an established practice in affective computing. The principal goal is to reduce unwanted subjective bias in the labelling process. Unfortunately, this leads to the key problem of identifying a ground truth for training the affect recognition system. This problem becomes more relevant in a sparsely-crossed annotation where each rater only labels a portion of the full dataset to ensure a manageable workload per rater. In this paper, we introduce a Multi-Rater Consensus Learning (MRCL) method which learns a representative affect recognition model that accounts for each rater's agreement with the other raters. MRCL combines a multitask learning (MTL) regularizer and a consensus loss. Unlike standard MTL, this approach allows the model to learn to predict each rater's label while explicitly accounting for the consensus among raters. We evaluated our approach on two different datasets based on spontaneous affective body movement expressions for pain behaviour detection and laughter type recognition respectively. The two naturalistic datasets were chosen for the different forms of labelling (different in affect, observation stimuli, and raters) that they together offer for evaluating our approach. Empirical results demonstrate that MRCL is effective for modelling affect from datasets with sparsely-crossed multi-rater annotation.

The 9TH AAAC Conference on Affective Computing and Intelligent Interaction 2021 was held in a virtual format in the fall of 2021. It was technically co-sponsored by the IEEE Computer Society and featured the recent work on Affective Computing. The six best papers from this conference were selected by the technical program chairs. They were invited to submit their extended version to be considered for this special section at the IEEE Transactions on Affective Computing. Each submission was reviewed by at least three expert reviewers and was evaluated in terms of overall contribution and the adequacy of the additional content to warrant a new article. This special section features five accepted submissions whose major contributions are summarized below.

Wearables integrating movement sonification can support body-perception changes and related physical activity; yet, we lack design principles for such sonifications. Through two mixed-methods studies, we investigate sound pitch and movement direction interaction effects on self-perception during squat exercises. We measured effects on body perception, affective quality of the experience, and actual and perceived movement, and compared them with two control conditions: no-sound and vibrotactile feedback. Results show that regardless of movement direction, ascending pitch enhances several body feelings and overall experience quality, while descending pitch increases movement acceleration. These effects were moderated by exercise physical demand. Sound and vibrotactile feedback enhanced flexibility and strength feelings, respectively, and contributed to exercise completion in different ways. Sound was perceived as an internal-to-body force while vibrotactile feedback was perceived as an external-to-body force. Feedback effects were stronger in people with lower fitness levels. We discuss results in terms of malleability of body perceptions and highlight opportunities to support demanding physical activity through wearable devices.

Touch exploration of fabric is used to evaluate its properties, and it could further be leveraged to understand a consumer’s sensory experience and preference so as to support them in real time to make careful clothing purchase decisions. In this paper, we open up opportunities to explore the use of technology to provide such support with our FabricTouch dataset, i.e., a multimodal dataset of fabric assessment touch gestures. The dataset consists of bilateral forearm movement and muscle activity data captured while 15 people explored 114 different garments in total to evaluate them according to 5 properties (warmth, thickness, smoothness, softness, and flexibility). The dataset further includes subjective ratings of the garments with respect to each property and ratings of pleasure experienced in exploring the garment through touch. We further report baseline work on automatic detection. Our results suggest that it is possible to recognise the type of fabric property that a consumer is exploring based on their touch behaviour. We obtained mean F1 score of 0.61 for unseen garments, for 5 types of fabric property. The results also highlight the possibility of additionally recognizing the consumer’s subjective rating of the fabric when the property being rated is known, mean F1 score of 0.97 for unseen subjects, for 3 rating levels.

Games can be powerful vehicles for gender identity exploration and self-reflection but are often subject to designers' biases including gender representation, limiting such opportunities. Using a game jam as a research-through-design method, alongside qualitative interviews with the creators, this paper explores how the process of creating games and the games themselves can facilitate exploration of and reflection on gender identity. We highlight aspects of identity people want to explore; how different game elements can support processes of exploring these, and what aspects are missing in games. Further, the process of creating and playing helped participants reflect on and reframe their understanding of gender regardless of their identity/experience. Finally, we reflect on the process of designing an inclusive jam around the topic of gender identity, which can be sensitive and divisive. Our work results in implications for the design of games and other potential tools for gender exploration.

Ninety percent of the 1.2 billion people who need assistive technology (AT) do not have access. Information seeking practices directly impact the ability of AT producers, procurers, and providers (AT professionals) to match a user's needs with appropriate AT, yet the AT marketplace is interdisciplinary and fragmented, complicating information seeking. We explored common limitations experienced by AT professionals when searching information to develop solutions for a diversity of users with multi-faceted needs. Through Template Analysis of 22 expert interviews, we find current search engines do not yield the necessary information, or appropriately tailor search results, impacting individuals’ awareness of products and subsequently their availability and the overall effectiveness of AT provision. We present value-based design implications to improve functionality of future AT-information seeking platforms, through incorporating smarter systems to support decision-making and need-matching whilst ensuring ethical standards for disability fairness remain.

Given the importance of affective touch in human interactions, technology designers are increasingly attempting to bring this modality to the core of interactive technology. Advances in haptics and touch-sensing technology have been critical to fostering interest in this area. In this survey, we review how affective touch is investigated to enhance and support the human experience with or through technology. We explore this question across three different research areas to highlight their epistemology, main findings, and the challenges that persist. First, we review affective touch technology through the human–computer interaction literature to understand how it has been applied to the mediation of human–human interaction and its roles in other human interactions particularly with oneself, augmented objects/media, and affect-aware devices. We further highlight the datasets and methods that have been investigated for automatic detection and interpretation of affective touch in this area. In addition, we discuss the modalities of affective touch expressions in both humans and technology in these interactions. Second, we separately review how affective touch has been explored in human–robot and real-human–virtual-human interactions where the technical challenges encountered and the types of experience aimed at are different. We conclude with a discussion of the gaps and challenges that emerge from the review to steer research in directions that are critical for advancing affective touch technology and recognition systems. In our discussion, we also raise ethical issues that should be considered for responsible innovation in this growing area.

Chronic pain is a prevalent condition that affects everyday life of people around the world. Protective behaviors (strategies that are naturally but unhelpfully adopted by people with chronic pain to cope with fear of pain in executing harmless everyday movements) can lead to further disability over time if not recognized and addressed appropriately. In this paper, we build on previous work on unimodal, activity-independent, time-continuous protective behavior detection (PBD) by focusing on the fusion of muscle activity and body movement modalities for characterizing both protective behavior and its physical activity context. We explore different fusion strategies based on consideration of the manner in which protective behavior influences muscle activity and overt body movement as well as the relationship between the two modalities. We evaluate the various strategies on the multimodal EmoPain dataset containing data from people with and without chronic pain engaged in physical activities that reflect everyday challenges for those with chronic pain. Our results show that a central (model-level) fusion approach leads to better PBD performance than input- and decision-level fusions, or unimodal approaches. We also show that additional use of attention mechanism, typifying shifts in attention characteristic of protective behavior, further improves the sensitivity of the model, i.e. detection of the positive class (which is the minority class). We analyze these results and suggest that fusion in modelling a motor condition should consider how emotional responses (fear of movement and pain in this case) triggered by a condition affect each of the given modalities and hence their contributions to the modelling task.

While there is growing interest in developing tech-nology to support pain assessment, pain-related self-management, and healthcare personalisation, there are currently no datasets on nonverbal pain behaviour in the context of functional activities. To address this gap, we introduce the EmoPain(at)Home dataset which consists of motion capture data and self-reported pain, worry, and confidence intensities captured from people with chronic pain. The data were recorded during self-selected functional activities in the home, e.g. vacuuming. We include analysis of the dataset as well as baseline classification of pain levels with average F1 score of 0.61 for two classes. We additionally discuss inclusivity considerations for capture of datasets in naturalistic settings, based on lessons learnt within our study.

For technology (like serious games) that aims to deliver interactive learning, it is important to address relevant mental experiences such as reflective thinking during problem solving. To facilitate research in this direction, we present the weDraw-1 Movement Dataset of body movement sensor data and reflective thinking labels for 26 children solving mathematical problems in unconstrained settings where the body (full or parts) was required to explore these problems. Further, we provide qualitative analysis of behaviours that observers used in identifying reflective thinking moments in these sessions. The body movement cues from our compilation informed features that led to average F1 score of 0.73 for binary classification of problem-solving episodes by reflective thinking based on Long Short-Term Memory neural networks. We further obtained 0.79 average F1 score for end-to-end classification, i.e., based on raw sensor data. Finally, the algorithms resulted in 0.64 average F1 score for subsegments of these episodes as short as 4 seconds. Overall, our results show the possibility of detecting reflective thinking moments from body movement behaviours of a child exploring mathematical concepts bodily, such as within serious game play.

Computational systems that process multiple affective states may benefit from explicitly considering the interaction between the states to enhance their recognition performance. This work proposes the combination of a multi-label classifier, Circular Classifier Chain (CCC), with a multimodal classifier, Fusion using a Semi-Naive Bayesian classifier (FSNBC), to include explicitly the dependencies between multiple affective states during the automatic recognition process. This combination of classifiers is applied to a virtual rehabilitation context of post-stroke patients. We collected data from post-stroke patients, which include finger pressure, hand movements, and facial expressions during ten longitudinal sessions. Videos of the sessions were labelled by clinicians to recognize four states: tiredness, anxiety, pain, and engagement. Each state was modelled by the FSNBC receiving the information of finger pressure, hand movements, and facial expressions. The four FSNBCs were linked in the CCC to exploit the dependency relationships between the states. The convergence of CCC was reached by 5 iterations at most for all the patients. Results (ROC AUC) of CCC with the FSNBC are over$0.940 \pm 0.045$($mean \pm std.\;deviation$) for the four states. Relationships of mutual exclusion between engagement and all the other states and co-occurrences between pain and anxiety were detected and discussed.

The problem of continuous emotion recognition has been the subject of several studies. The proposed affective computing approaches employ sequential machine learning algorithms for improving the classification stage, accounting for the time ambiguity of emotional responses. Modeling and predicting the affective state over time is not a trivial problem because continuous data labeling is costly and not always feasible. This is a crucial issue in real-life applications, where data labeling is sparse and possibly captures only the most important events rather than the typical continuous subtle affective changes that occur. In this work, we introduce a framework from the machine learning literature called Multiple Instance Learning, which is able to model time intervals by capturing the presence or absence of relevant states, without the need to label the affective responses continuously (as required by standard sequential learning approaches). This choice offers a viable and natural solution for learning in a weakly supervised setting, taking into account the ambiguity of affective responses. We demonstrate the reliability of the proposed approach in a gold-standard scenario and towards real-world usage by employing an existing dataset (DEAP) and a purposely built one (Consumer). We also outline the advantages of this method with respect to standard supervised machine learning algorithms.

We are at a tipping point for digital communication: moving beyond 'ways of seeing' to include 'ways of feeling'.Much as optical technologies transformed sight and the visual (from the telescope and microscope to Google Glass), the rapid expansion in digital touch technologies is set to reconfigure touch and the tactile in significant ways.Advances in haptics, virtual reality, and physiological sensing provide new sensory ways of communicating, as well as new ways to capture the quality of touch.These state-of-the-art digital touch technologies promise to supplement, heighten, extend and reconfigure how people communicate.They are reshaping what and who can be touched, as well as when and how they can be touched, changing existing forms of communication and giving rise to changes in co-located and remote communication between humans, and between humans and robots.These developments sit alongside social discourses of concern and loss, with the digital being associated with the removal of touch from the material sensory landscape (Jewitt et al., 2020).Yet, in our current climate of social distancing and disengaging with touch in our everyday interactions, the promise of the digital becomes increasingly appealing and significant, in re-enabling touch possibilities.As the global population emerges from and comes to terms with their experiences of the COVID-19 pandemic, these questions become all the more significant.The breadth and interdisciplinary interest in this growing field-across designers, artists, computer scientists, engineers, psychologists and social scientists, with interests in robotics and touch, affective computing, wearables, and digital installations -brings attention to the growing need to engage with 'social' aspects of digital touch, moving beyond technologies and physiological foci to engage with touch practices (Jewitt et al., 2021).We argue for the need to think about touch beyond the physiological act of sensing and perceiving to a more rich and nuanced interpretation of touch that takes account of its emotional and psychological significance, the social, cultural and historical evolution of touch practices in human communication, and approaches to touch of the "lived, social body as a site of meaning-making, where the skin acts as both a boundary between and a point of connection with others" (Karpashevich et al.).Given this landscape, this special issue addresses timely and important questions around the need to think about touch in different ways.This provides a new direction in the field that seeks to address the complexity of human touch and interrogates the limitation of today's haptic devices.Papers in this special issue contribute to understanding this gap by engaging with socio-cultural issues around digital touch communication and identifying new ways to study touch, specifically aiming to explore the nuance and subtlety of touch and an expanded view of touch.

Despite the importance of touch in human–human relations, research in affective tactile practices is in its infancy, lacking in-depth understanding needed to inform the design of remote digital touch communication. This article reports two qualitative studies that explore tactile affective communication in specific social contexts, and the bi-directional creation, sending and interpretation of digital touch messages using a purpose-built research tool, the Tactile Emoticon. The system comprises a pair of remotely connected mitts, which enable users in different locations to communicate through tactile messages, by orchestrating duration and level of three haptic sensations: vibration, pressure and temperature. Qualitative analysis shows the nuanced ways in which 68 participants configured these elements to make meaning from touch messages they sent and received. It points to the affect and emotion of touch, its sensoriality and ambiguity, the significance of context, social norms and expectations of touch participants. Findings suggest key design considerations for digital touch communication, where the emphasis shifts from generating ‘recognizable touches’ to tools that allow people to shape their touches and establish common understanding about their meaning.

Breakthroughs in unsupervised domain adaptation (uDA) can help in adapting models from a label-rich source domain to unlabeled target domains. Despite these advancements, there is a lack of research on how uDA algorithms, particularly those based on adversarial learning, can work in distributed settings. In real-world applications, target domains are often distributed across thousands of devices, and existing adversarial uDA algorithms – which are centralized in nature – cannot be applied in these settings. To solve this important problem, we introduce FRuDA: an end-to-end framework for distributed adversarial uDA. Through a careful analysis of the uDA literature, we identify the design goals for a distributed uDA system and propose two novel algorithms to increase adaptation accuracy and training efficiency of adversarial uDA in distributed settings. Our evaluation of FRuDA with five image and speech datasets show that it can boost target domain accuracy by up to 50% and improve the training efficiency of adversarial uDA by at least$11\times$.

Comfort is a subjective experience that people attend to in everyday life including in cars where they are constrained in movement. Could intelligent cars sense their comfort levels for the purpose of maximizing it? To address this, first, we present a new dataset (available on request) of physical measures (skin temperature, blood volume pulse, electrodermal activity, and motion capture) and subjective thermal, sitting, and mental relaxation experience variables captured in semi-ecological settings in a car. Second, we provide an in-depth analysis of the relationship between passengers’ thermal experiences and physiological responses in the collected data. Our findings highlight complex duality in the relationship of thermal experience with heart rate variability and skin temperature variability. We discuss the practical implications that this may have for designing machine learning architectures for automatic detection of thermal discomfort.

Data from multiple sensors can boost the automatic recognition of multiple affective states in a multilabel and multimodal recognition system. At any time, the streaming from any of the contributing sensors can be missing. This work proposes a method for dealing with a missing sensor in a multilabel and multimodal automatic affective states recognition system. The proposed method, called Hot Deck using Conditional Probability Tables (HD-CPT), is incorporated into a multimodal affective state recognition system for compensating the loss of a sensor using the recorded historical information of the sensor and its interaction with the other available sensors. In this work, we consider a multilabel classifier, named Circular Classifier Chain, for the automatic recognition of four states: tiredness, anxiety, pain, and engagement; combined with a multimodal classifier based on three sensors: fingers pressure, hand movements, and facial expressions; which was adapted for coping with the problem of a missing sensor in a virtual rehabilitation platform for post-stroke patients. A dataset of five post-stroke patients who attended ten longitudinal rehabilitation sessions was used for the evaluation. The inclusion of HD-CPT compensated for the loss of one sensor with results above those obtained with only the remaining sensors available. HD-CPT prevents the system from collapsing when a sensor fails, providing continuity of operation with results that attenuate the loss of the sensor. The proposed method HD-CPT can provide robustness for the naturalistic everyday use of an affective states recognition system.

Orientation and mobility (O&M) training provides essential skills and techniques for safe and independent mobility for blind and partially sighted (BPS) people. The demand for O&M training is increasing as the number of people living with vision impairment increases. Despite the growing portfolio of HCI research on assistive technologies (AT), few studies have examined the experiences of BPS people during O&M training, including the use of technology to aid O&M training. To address this gap, we conducted semi-structured interviews with 20 BPS people and 8 Mobility and Orientation Trainers (MOT). The interviews were thematically analysed and organised into four overarching themes discussing factors influencing the self-efficacy belief of BPS people: Tools and Strategies for O&M training, Technology Use in O&M Training, Changing Personal and Social Circumstances, and Social Influences. We further highlight opportunities for combinations of multimodal technologies to increase access to and effectiveness of O&M training.

Negative body perceptions are a major predictor of physical inactivity, a serious health concern. Sensory feedback can be used to alter such body perceptions; movement sonification, in particular, has been suggested to affect body perception and levels of physical activity (PA) in inactive people. We investigated how metaphorical sounds impact body perception and PA. We report two qualitative studies centered on performing different strengthening/flexibility exercises using SoniBand, a wearable that augments movement through different sounds. The first study involved physically active participants and served to obtain a nuanced understanding of the sonifications’ impact. The second, in the home of physically inactive participants, served to identify which effects could support PA adherence. Our findings show that movement sonification based on metaphors led to changes in body perception (e.g., feeling strong) and PA (e.g., repetitions) in both populations, but effects could differ according to the existing PA-level. We discuss principles for metaphor-based sonification design to foster PA.

No abstract available.

In chronic pain rehabilitation, physiotherapists adapt physical activity to patients’ performance based on their expression of protective behavior, gradually exposing them to feared but harmless and essential everyday activities. As rehabilitation moves outside the clinic, technology should automatically detect such behavior to provide similar support. Previous works have shown the feasibility of automatic protective behavior detection (PBD) within a specific activity. In this article, we investigate the use of deep learning for PBD across activity types, using wearable motion capture and surface electromyography data collected from healthy participants and people with chronic pain. We approach the problem by continuously detecting protective behavior within an activity rather than estimating its overall presence. The best performance reaches mean F1 score of 0.82 with leave-one-subject-out cross validation. When protective behavior is modeled per activity type, performance achieves a mean F1 score of 0.77 for bend-down, 0.81 for one-leg-stand, 0.72 for sit-to-stand, 0.83 for stand-to-sit, and 0.67 for reach-forward. This performance reaches excellent level of agreement with the average experts’ rating performance suggesting potential for personalized chronic pain management at home. We analyze various parameters characterizing our approach to understand how the results could generalize to other PBD datasets and different levels of ground truth granularity.

Interacting with natural environments such as parks and the countryside improves health and wellbeing. These spaces allow for exercise, relaxation, socialising and exploring nature, however, they are often not used by blind and partially sighted people (BPSP). To better understand the needs of BPSP for outdoor leisure experience and barriers encountered in planning, accessing and engaging with natural environments, we conducted an exploratory qualitative online survey (22 BPSP), semi-structured interviews (20 BPSP) and a focus group (9 BPSP; 1 support worker). We also explored how current technologies support park experiences for BPSP. Our findings identify common barriers across the stages of planning (e.g. limited accessible information about parks), accessing (e.g. poor wayfinding systems), engaging with and sharing leisure experiences. Across all stages (PLan, Access, Engage, Share) we found a common theme of Contribute. BPSP wished to co-plan their trip, contribute to ways of helping others access a place, develop multisensory approaches to engaging in their surroundings and share their experiences to help others. In this paper, we present the initial work supporting the development of a framework for understanding the leisure experiences of BPSP. We explore this theme of contribution and propose a framework where this feeds into each of the stages of leisure experience, resulting in the proposed, PLACES framework (PLan, Access, Contribute, Engage, Share), which aims to provide a foundation for future research on accessibility and outdoor leisure experiences for BPSP and people with disabilities.

The EmoPain 2020 Challenge is the first international competition aimed at creating a uniform platform for the comparison of multi-modal machine learning and multimedia processing methods of chronic pain assessment from human expressive behaviour, and also the identification of pain-related behaviours. The objective of the challenge is to promote research in the development of assistive technologies that help improve the quality of life for people with chronic pain via real-time monitoring and feedback to help manage their condition and remain physically active. The challenge also aims to encourage the use of the relatively underutilised, albeit vital bodily expression signals for automatic pain and pain-related emotion recognition. This paper presents a description of the challenge, competition guidelines, bench-marking dataset, and the baseline systems' architecture and performance on the Challenge's three sub-tasks: pain estimation from facial expressions, pain recognition from multimodal movement, and protective movement behaviour detection.

This paper introduces a new dataset, Libri-Adapt, to support unsupervised domain adaptation research on speech recognition models. Built on top of the LibriSpeech corpus, Libri-Adapt contains 7200 hours of English speech recorded on mobile and embedded-scale microphones, and spans 72 different domains that are representative of the challenging practical scenarios encountered by ASR models. More specifically, Libri-Adapt facilitates the study of domain shifts in ASR models caused by a) different acoustic environments, b) variations in speaker accents, c) previously unexplored factors such as heterogeneity in the hardware and platform software of the microphones, and d) a combination of the aforementioned three shifts. We also provide a number of baseline results quantifying the impact of these domain shifts on the Mozilla DeepSpeech2 ASR model.

We propose a novel neural network architecture, named the Global Workspace Network (GWN), which addresses the challenge of dynamic and unspecified uncertainties in multimodal data fusion. Our GWN is a model of attention across modalities and evolving through time, and is inspired by the well-established Global Workspace Theory from the field of cognitive science. The GWN achieved average F1 score of 0.92 for discrimination between pain patients and healthy participants and average F1 score = 0.75 for further classification of three pain levels for a patient, both based on the multimodal EmoPain dataset captured from people with chronic pain and healthy people performing different types of exercise movements in unconstrained settings. In these tasks, the GWN significantly outperforms the typical fusion approach of merging by concatenation. We further provide extensive analysis of the behaviour of the GWN and its ability to address uncertainties (hidden noise) in multimodal data.

Multimodal interfaces pose the challenge of dealing with the multi-ple interactive time-scales characterizing human behavior. To dothis, innovative models and time-adaptive technologies are needed,operating at multiple time-scales and adopting a multi-layered ap-proach. The first International Workshop on Multi-Scale MovementTechnologies, hosted virtually during the 22nd ACM InternationalConference on Multimodal Interaction, is aimed at providing re-searchers from different areas with the opportunity to discuss thistopic. This paper summarizes the activities of the workshop andthe accepted papers

Unsupervised domain adaptation using adversarial learning has shown promise in adapting speech models from a labeled source domain to an unlabeled target domain. However, prior works make a strong assumption that the label spaces of source and target domains are identical, which can be easily violated in real-world conditions. We present AMLS, an end-to-end architecture that performs Adaptation under Mismatched Label Spaces using two weighting schemes to separate shared and private classes in each domain. An evaluation on three speech adaptation tasks, namely gender, microphone, and emotion adaptation, shows that AMLS provides significant accuracy gains over baselines used in speech and vision adaptation tasks. Our contribution paves the way for applying UDA to speech models in unconstrained settings with no assumptions on the source and target label spaces.

Convolutional neural networks (CNNs) offer great machine learning performance over a range of applications, but their operation is hard to interpret, even for experts. Various explanation algorithms have been proposed to address this issue, yet limited research effort has been reported concerning their user evaluation. In this paper, we report on an online between-group user study designed to evaluate the performance of "saliency maps" - a popular explanation algorithm for image classification applications of CNNs. Our results indicate that saliency maps produced by the LRP algorithm helped participants to learn about some specific image features the system is sensitive to. However, the maps seem to provide very limited help for participants to anticipate the network's output for new images. Drawing on our findings, we highlight implications for design and further research on explainable AL In particular, we argue the HCI and AI communities should look beyond instance-level explanations.

Clinicians tailor intervention in chronic pain rehabilitation to movement related self-efficacy (MRSE). This motivates us to investigate automatic MRSE estimation in this context towards the development of technology that is able to provide appropriate support in the absence of a clinician. We first explored clinical observer estimation, which showed that body movement behaviours, rather than facial expressions or engagement behaviours, were more pertinent to MRSE estimation during physical activity instances. Based on our findings, we built a system that estimates MRSE from bodily expressions and bodily muscle activity captured using wearable sensors. Our results (F1 scores of 0.95 and 0.78 in two physical exercise types) provide evidence of the feasibility of automatic MRSE estimation to support chronic pain physical rehabilitation. We further explored automatic estimation of MRSE with a reduced set of low-cost sensors to investigate the possibility of embedding such capabilities in ubiquitous wearable devices to support functional activity. Our evaluation for both exercise and functional activity resulted in F1 score of 0.79. This result suggests the possibility of (and calls for more studies on) MRSE estimation during everyday functioning in ubiquitous settings. We provide a discussion of the implication of our findings for relevant areas.

Virtual rehabilitation environments may afford greater patient personalization if they could harness the patient's affective state. Four states: anxiety, pain, engagement and tiredness (either physical or psychological), were hypothesized to be inferable from observable metrics of hand location and gripping strength-relevant for rehabilitation. Contributions are; (a) multiresolution classifier built from Semi-Naïve Bayesian classifiers, and (b) establishing predictive relations for the considered states from the motor proxies capitalizing on the proposed classifier with recognition levels sufficient for exploitation. 3D hand locations and gripping strength streams were recorded from 5 post-stroke patients whilst undergoing motor rehabilitation therapy administered through virtual rehabilitation along 10 sessions over 4 weeks. Features from the streams characterized the motor dynamics, while spontaneous manifestations of the states were labelled from concomitant videos by experts for supervised classification. The new classifier was compared against baseline support vector machine (SVM) and random forest (RF) with all three exhibiting comparable performances. Inference of the aforementioned states departing from chosen motor surrogates appears feasible, expediting increased personalization of virtual motor neurorehabilitation therapies.

Automatically monitoring and quantifying stress-induced thermal dynamic information in real-world settings is an extremely important but challenging problem. In this paper, we explore whether we can use mobile thermal imaging to measure the rich physiological cues of mental stress that can be deduced from a person's nose temperature. To answer this question we build i) a framework for monitoring nasal thermal variable patterns continuously and ii) a novel set of thermal variability metrics to capture a richness of the dynamic information. We evaluated our approach in a series of studies including laboratory-based psychosocial stress-induction tasks and real-world factory settings. We demonstrate our approach has the potential for assessing stress responses beyond controlled laboratory settings.

Physical inactivity is an increasing problem. It has been linked to psychological and emotional barriers related to the perception of one's body, such as physical capabilities. It remains a challenge to design technologies to increase physical activity in inactive people. We propose the use of a sound interactive system where inputs from movement sensors integrated in shoes are transformed into sounds that evoke body sensations at a metaphorical level. Our user study investigates the effects of various gesture-sound mappings on the perception of one's body and its movement qualities (e.g. being flexible or agile), the related emotional state and movement patterns, when people performed two exercises, walking and thigh stretch. The results confirm the effect of the “metaphor” conditions vs. the control conditions in feelings of body weight; feeling less tired and more in control; or being more comfortable, motivated, and happier. These changes linked to changes in affective state and body movement. We discuss the results in terms of how acting upon body perception and affective states through sensory feedback may in turn enhance physical activity, and the opportunities opened by our findings for the design of wearable technologies and interventions in inactive populations.

The automatic recognition of multiple affective states can be enhanced if the underpinning computational models explicitly consider the interactions between the states. This work proposes a computational model that incorporates the dependencies between four states (tiredness, anxiety, pain, and engagement)known to appear in virtual rehabilitation sessions of post-stroke patients, to improve the automatic recognition of the patients' states. A dataset of five stroke patients which includes their fingers' pressure (PRE), hand movements (MOV)and facial expressions (FAE)during ten sessions of virtual rehabilitation was used. Our computational proposal uses the Semi-Naive Bayesian classifier (SNBC)as base classifier in a multiresolution approach to create a multimodal model with the three sensors (PRE, MOV, and FAE)with late fusion using SNBC (FSNB classifier). There is a FSNB classifier for each state, and they are linked in a circular classifier chain (CCC)to exploit the dependency relationships between the states. Results of CCC are over 90% of ROC AUC for the four states. Relationships of mutual exclusion between engagement and all the other states and some co-occurrences between pain and anxiety for the five patients were detected. Virtual rehabilitation platforms that incorporate the automatic recognition of multiple patient's states could leverage intelligent and empathic interactions to promote adherence to rehabilitation exercises.

When developing interactive systems for children, such as serious games in the context of educational technology, it is important to take into account and address relevant cognitive and emotional child's experiences that may influence learning outcomes. Some works were done to analyze and automatically recognize these cognitive and affective states from nonverbal expressive behaviors. However, there is a lack of knowledge about visually impaired children and their body language to convey those states during learning tasks. In this paper, we present an analysis of nonverbal expressive behaviors of both blind and low-vision children, aiming at understanding what type of body communication can be an indicator of two cognitive states: engagement and confidence. In the study we consider the data collected along the EU-ICT H2020 weDRAW Project, while children were asked to solve mathematical tasks with their body. For such a dataset, we propose a list of 31 nonverbal behaviors, annotated both by rehabilitators used to work with visually impaired children and by naive observers. In the last part of the paper, we propose a preliminary study on automatic recognition of engagement and confidence states from 2D positional data. The classification results are up to 0.71 (F-score) on a three-class classification task.

Increasingly popular, long-distance running events (LDRE) attract not just runners but an exponentially increasing number of spectators. Due to the long duration and broad geographic spread of such events, interactions between them are limited to brief moments when runners (R) pass by their supporting spectators (S). Current technology is limited in its potential for supporting interactions and mainly measures and displays basic running information to spectators who passively consume it. In this paper, we conducted qualitative studies for an in-depth understanding of the R&S' shared experience during LDRE and how technology can enrich this experience. We propose a two-layer DyPECS framework, highlighting the rich dynamics of the R&S multi-faceted running journey and of their micro-encounters. DyPECS is enriched by the findings from our in depth qualitative studies. We finally present design implications for the multi-facet co-experience of R&S during LDRE.

Supporting exercise adherence through technology remains an important HCI challenge. Recent works showed that altering walking sounds leads people perceiving themselves as thinner/lighter, happier and walking more dynamically. While this novel approach shows potential for physical activity, it raises critical questions impacting technology design. We ran two studies in the context of exertion (gym-step, stairs-climbing) to investigate how individual factors impact the effect of sound and the duration of the after-effects. The results confirm that the effects of sound in body-perception occur even in physically demanding situations and through ubiquitous wearable devices. We also show that the effect of sound interacted with participants' body weight and masculinity/femininity aspirations, but not with gender. Additionally, changes in body-perceptions did not hold once the feedback stopped; however, body-feelings or behavioural changes appeared to persist for longer. We discuss the results in terms of malleability of body-perception and highlight opportunities for supporting exercise adherence.

In chronic pain physical rehabilitation, physiotherapists adapt exercise sessions according to the movement behavior of patients. As rehabilitation moves beyond clinical sessions, technology is needed to similarly assess movement behaviors and provide such personalized support. In this paper, as a first step, we investigate automatic detection of protective behavior (movement behavior due to pain-related fear or pain) based on wearable motion capture and electromyography sensor data. We investigate two recurrent networks (RNN) referred to as stacked-LSTM and dual-stream LSTM, which we compare with related deep learning (DL) architectures. We further explore data augmentation techniques and additionally analyze the impact of segmentation window lengths on detection performance. The leading performance of 0.815 mean F1 score achieved by stacked-LSTM provides important grounding for the development of wearable technology to support chronic pain physical rehabilitation during daily activities.

Unsupervised domain adaptation is emerging as a powerful technique to improve the generalizability of deep learning models to new image domains without using any labeled data in the target domain. In the literature, solutions which perform cross-domain feature-matching (e.g., ADDA), pixel-matching (CycleGAN), and combination of the two (e.g., CyCADA) have been proposed for unsupervised domain adaptation. Many of these approaches make a strong assumption that the source and target label spaces are the same, however in the real-world, this assumption does not hold true. In this paper, we propose a novel solution, FlexAdapt, which extends the state-of-the-art unsupervised domain adaptation approach of CyCADA to scenarios where the label spaces in source and target domains are only partially overlapped. Our solution beats a number of state-of-the-art baseline approaches by as much as 29% in some scenarios, and represent a way forward for applying domain adaptation techniques in the real world.

Mobile and embedded devices are increasingly using microphones and audio-based computational models to infer user context. A major challenge in building systems that combine audio models with commodity microphones is to guarantee their accuracy and robustness in the real-world. Besides many environmental dynamics, a primary factor that impacts the robustness of audio models is microphone variability. In this work, we propose Mic2Mic - a machine-learned system component - which resides in the inference pipeline of audio models and at real-time reduces the variability in audio data caused by microphone-specific factors. Two key considerations for the design of Mic2Mic were: a) to decouple the problem of microphone variability from the audio task, and b) put minimal burden on end-users to provide training data. With these in mind, we apply the principles of cycle-consistent generative adversarial networks (CycleGANs) to learn Mic2Mic using unlabeled and unpaired data collected from different microphones. Our experiments show that Mic2Mic can recover between 66% to 89% of the accuracy lost due to microphone variability for two common audio tasks.

Although clinical best practice suggests that affect awareness could enable more effective technological support for physical rehabilitation through personalisation to psychological needs, designers need to consider what affective states matter, and how they should be tracked and addressed. In this article, we set the standard by analysing how the major affective factors in chronic pain (pain, fear/anxiety, and low/depressed mood) interfere with everyday physical functioning. Further, based on discussion of the modality that should be used to track these states to enable technology to address them, we investigated the possibility of using movement behaviour to automatically detect the states. Using two body movement datasets on people with chronic pain, we show that movement behaviour enables very good discrimination between two emotional distress levels (F1=0.86), and three pain levels (F1=0.9). Performance remained high (F1=0.78 for two pain levels) with a reduced set of movement sensors. Finally, in an overall discussion, we suggest how technology-provided encouragement and awareness can be personalised given the capability to automatically monitor the relevant states, towards addressing the barriers that they pose. In addition, we highlight movement behaviour features to be tracked to provide technology with information necessary for such personalisation.

We introduce Deep Thermal Imaging, a new approach for close-range automatic recognition of materials to enhance the understanding of people and ubiquitous technologies of their proximal environment. Our approach uses a low-cost mobile thermal camera integrated into a smartphone to capture thermal textures. A deep neural network classifies these textures into material types. This approach works effectively without the need for ambient light sources or direct contact with materials. Furthermore, the use of a deep learning network removes the need to handcraft the set of features for different materials. We evaluated the performance of the system by training it to recognize 32 material types in both indoor and outdoor environments. Our approach produced recognition accuracies above 98% in 14,860 images of 15 indoor materials and above 89% in 26,584 images of 17 outdoor materials. We conclude by discussing its potentials for real-time use in HCI applications and future directions.

We propose DeepBreath, a deep learning model which automatically recognises people's psychological stress level (mental overload) from their breathing patterns. Using a low cost thermal camera, we track a person's breathing patterns as temperature changes around his/her nostril. The paper's technical contribution is threefold. First of all, instead of creating handcrafted features to capture aspects of the breathing patterns, we transform the uni-dimensional breathing signals into two dimensional respiration variability spectrogram (RVS) sequences. The spectrograms easily capture the complexity of the breathing dynamics. Second, a spatial pattern analysis based on a deep Convolutional Neural Network (CNN) is directly applied to the spectrogram sequences without the need of hand-crafting features. Finally, a data augmentation technique, inspired from solutions for over-fitting problems in deep learning, is applied to allow the CNN to learn with a small-scale dataset from short-term measurements (e.g., up to a few hours). The model is trained and tested with data collected from people exposed to two types of cognitive tasks (Stroop Colour Word Test, Mental Computation test) with sessions of different difficulty levels. Using normalised self-report as ground truth, the CNN reaches 84.59% accuracy in discriminating between two levels of stress and 56.52% in discriminating between three levels. In addition, the CNN outperformed powerful shallow learning methods based on a single layer neural network. Finally, the dataset of labelled thermal images will be open to the community.

While most rehabilitation technologies target situated exercise sessions and associated performance metrics, physiotherapists recommend physical activities that are integrated with everyday functioning. We conducted a 1-2 week home study to explore how people with chronic pain use wearable technology that senses and sonifies movement (i.e., movement mapped to sound in real-time) to do functional activity (e.g., loading the dishwasher). Our results show that real-time movement sonification led to an increased sense of control during challenging everyday tasks. Sonification calibrated to functional activity facilitated application of pain management techniques such as pacing. When calibrated to individual needs, sonification enabled serendipitous discovery of physical capabilities otherwise obscured by a focus on pain or a dysfunctional proprioceptive system. A physiotherapist was invited to comment on the implications of our findings. We conclude by discussing opportunities provided by wearable sensing technology to enable better functioning, the ultimate goal of physical rehabilitation.

The International Workshop on Multimodal Interaction for Education aims at investigating how multimodal interactive systems, firmly grounded on psychophysical, psychological, and pedagogical bases, can be designed, developed, and exploited for enhancing teaching and learning processes in different learning environments, with a special focus on children in the classroom. Whilst the usage of multisensory technologies in the education area is rapidly expanding, the need for solid scientific bases, design guidelines, and appropriate procedures for evaluation is emerging. Moreover, the introduction of multimodal interactive systems in the learning environment needs to develop at the same time suitable pedagogical paradigms. This workshop aims at bringing together researchers and practitioners from different disciplines, including pedagogy, psychology, psychophysics, and computer science - with a particular focus on human-computer interaction, affective computing, and social signal processing - to discuss such challenges under a multidisciplinary perspective. The workshop is partially supported by the EU-H2020-ICT Project weDRAW (http://www.wedraw.eu).

In this paper, I argue that the affective body is underused in the design of interactive technology despite what it has to offer. Whilst the literature shows it to be a powerful affective communication channel, it is often ignored in favor of the more commonly studied facial and vocal expression modalities. This is despite it being as informative and in some situations even more reliable than the other affective channels. In addition, due to the proliferation of increasingly cheaper and ubiquitous movement sensing technologies, the regulatory affective functions of the body could open new possibilities in various application areas. In this paper, after presenting a brief summary of the opportunities that the affective body offers to technology designers, I will use the case of physical rehabilitation to discuss how its use could lead to interesting new solutions and more effective therapies.

In self-directed chronic pain physical rehabilitation it is important that the individual can progress as physical capabilities and confidence grow. However, people with chronic pain often struggle to pass what they have identified as safe boundaries. At the same time, over-activity due to the desire to progress fast or function more normally, may lead to setbacks. We investigate how musically-informed movement sonification can be used as an implicit mechanism to both avoid overdoing and facilitate progress during stretching exercises. We sonify an end target-point in a stretch exercise, using a stable sound (i.e., where the sonification is musically resolved) to encourage movements ending and an unstable sound (i.e., musically unresolved) to encourage continuation. Results on healthy participants show that instability leads to progression further beyond the target-point while stability leads to a smoother stop beyond this point. We conclude discussing how these findings should generalize to the CP population.

Strategies are necessary to mitigate the impact of unexpected behavior in collaborative robotics, and research to develop solutions is lacking. Our aim here was to explore the benefits of an affective interaction, as opposed to a more efficient, less error prone but non-communicative one. The experiment took the form of an omelet-making task, with a wide range of participants interacting directly with BERT2, a humanoid robot assistant. Having significant implications for design, results suggest that efficiency is not the most important aspect of performance for users; a personable, expressive robot was found to be preferable over a more efficient one, despite a considerable trade off in time taken to perform the task. Our findings also suggest that a robot exhibiting human-like characteristics may make users reluctant to `hurt its feelings'; they may even lie in order to avoid this.

We present RealPen, an augmented stylus for capacitive tablet screens that recreates the physical sensation of writing on paper with a pencil, ball-point pen or marker pen. The aim is to create a more engaging experience when writing on touch surfaces, such as screens of tablet computers. This is achieved by regenerating the friction-induced oscillation and sound of a real writing tool in contact with paper. To generate realistic tactile feedback, our algorithm analyzes the frequency spectrum of the friction oscillation generated when writing with traditional tools, extracts principal frequencies, and uses the actuator's frequency response profile for an adjustment weighting function. We enhance the realism by providing the sound feedback aligned with the writing pressure and speed. Furthermore, we investigated the effects of superposition and fluctuation of several frequencies on human tactile perception, evaluated the performance of RealPen, and characterized users' perception and preference of each feedback type.

Chronic (persistent) pain (CP) affects 1 in 10 adults; clinical resources are insufficient, and anxiety about activity restricts lives. Technological aids monitor activity but lack necessary psychological support. This article proposes a new sonification framework, Go-with-the-Flow, informed by physiotherapists and people with CP. The framework proposes articulation of user-defined sonified exercise spaces (SESs) tailored to psychological needs and physical capabilities that enhance body and movement awareness to rebuild confidence in physical activity. A smartphone-based wearable device and a Kinect-based device were designed based on the framework to track movement and breathing and sonify them during physical activity. In control studies conducted to evaluate the sonification strategies, people with CP reported increased performance, motivation, awareness of movement, and relaxation with sound feedback. Home studies, a focus group, and a survey of CP patients conducted at the end of a hospital pain management session provided an in-depth understanding of how different aspects of the SESs and their calibration can facilitate self-directed rehabilitation and how the wearable version of the device can facilitate transfer of gains from exercise to feared or demanding activities in real life. We conclude by discussing the implications of our findings on the design of technology for physical rehabilitation.

Pain-related emotions are a major barrier to effective self rehabilitation in chronic pain. Automated coaching systems capable of detecting these emotions are a potential solution. This paper lays the foundation for the development of such systems by making three contributions. First, through literature reviews, an overview of how pain is expressed in chronic pain and the motivation for detecting it in physical rehabilitation is provided. Second, a fully labelled multimodal dataset (named `EmoPain') containing high resolution multiple-view face videos, head mounted and room audio signals, full body 3D motion capture and electromyographic signals from back muscles is supplied. Natural unconstrained pain related facial expressions and body movement behaviours were elicited from people with chronic pain carrying out physical exercises. Both instructed and non-instructed exercises were considered to reflect traditional scenarios of physiotherapist directed therapy and home-based self-directed therapy. Two sets of labels were assigned: level of pain from facial expressions annotated by eight raters and the occurrence of six pain-related body behaviours segmented by four experts. Third, through exploratory experiments grounded in the data, the factors and challenges in the automated recognition of such expressions and behaviour are described, the paper concludes by discussing potential avenues in the context of these findings also highlighting differences for the two exercise scenarios addressed.

Automatic continuous affective state prediction from naturalistic facial expression is a very challenging research topic but very important in human-computer interaction. One of the main challenges is modeling the dynamics that characterize naturalistic expressions. In this paper, a novel two-stage automatic system is proposed to continuously predict affective dimension values from facial expression videos. In the first stage, traditional regression methods are used to classify each individual video frame, while in the second stage, a time-delay neural network (TDNN) is proposed to model the temporal relationships between consecutive predictions. The two-stage approach separates the emotional state dynamics modeling from an individual emotional state prediction step based on input features. In doing so, the temporal information used by the TDNN is not biased by the high variability between features of consecutive frames and allows the network to more easily exploit the slow changing dynamics between emotional states. The system was fully tested and evaluated on three different facial expression video datasets. Our experimental results demonstrate that the use of a two-stage approach combined with the TDNN to take into account previously classified frames significantly improves the overall performance of continuous emotional state estimation in naturalistic facial expressions. The proposed approach has won the affect recognition sub-challenge of the Third International Audio/Visual Emotion Recognition Challenge.

Laughter is a very interesting non-verbal human vocalization. It is classified as a semi voluntary behavior despite being a direct form of social interaction, and can be elicited by a variety of very different stimuli, both cognitive and physical. Automatic laughter detection, analysis and classification will boost progress in affective computing, leading to the development of more natural human-machine communication interfaces. Surface Electromyography (sEMG) on abdominal muscles or invasive EMG on the larynx show potential in this direction, but these kinds of EMG-based sensing systems cannot be used in ecological settings due to their size, lack of reusability and uncomfortable setup. For this reason, they cannot be easily used for natural detection and measurement of a volatile social behavior like laughter in a variety of different situations. We propose the use of miniaturized, wireless, dry-electrode sEMG sensors on the neck for the detection and analysis of laughter. Even if with this solution the activation of specific larynx muscles cannot be precisely measured, it is possible to detect different EMG patterns related to larynx function. In addition, integrating sEMG analysis on a multisensory compact system positioned on the neck would improve the overall robustness of the whole sensing system, enabling the synchronized measure of different characteristics of laughter, like vocal production, head movement or facial expression; being at the same time less intrusive, as the neck is normally more accessible than abdominal muscles. In this paper, we report laughter discrimination rate obtained with our system depending on different conditions.

Mimicry and laughter are two social signals displaying affiliation among people. To date, however, their relationship remains uninvestigated and relatively unexploited in designing the behaviour of robots and virtual characters. This paper presents an experiment aimed at examining how laughter and mimicry are related. The hypothesis is that hand movements a person produces during a laughter episode are mimicked through equivalent or other hand movements other participants in the interaction produce when they laugh. To investigate this, we analysed mimicry at two levels of specificity during laughter and non-laughter periods in a playful triadic social interaction. Changes in mimicry rates over the whole interaction were analysed as well as possible leader-follower relationships. Results show that hand movement rates were varied and strongly dependent on group. Even though hand movement are more frequent during laughter, mimicry does not increase. Mimicry levels, however, increase over the course of a session indicating that familiarity and comfort may increase emotional contagion.

People with chronic musculoskeletal pain would benefit from technology that provides run-time personalized feedback and help adjust their physical exercise plan. However, increased pain during physical exercise, or anxiety about anticipated pain increase, may lead to setback and intensified sensitivity to pain. Our study investigates the possibility of detecting pain levels from the quality of body movement during two functional physical exercises. By analyzing recordings of kinematics and muscle activity, our feature optimization algorithms and machine learning techniques can automatically discriminate between people with low level pain and high level pain and control participants while exercising. Best results were obtained from feature set optimization algorithms: 94% and 80% for the full trunk flexion and sit-to-stand movements respectively using Support Vector Machines. As depression can affect pain experience, we included participants' depression scores on a standard questionnaire and this improved discrimination between the control participants and the people with pain when Random Forests were used.

Studying tactile experience is important and timely, considering how this channel is being harnessed both in terms of human interaction and for technological developments that rely on it to enhance experience of products and services. Research into tactile experience to date is present mostly within the social context, but there are not many studies on the understanding of tactile experience in interaction with objects. In this paper, we use textiles as a case study to investigate how we can get people to talk about this experience, and to understand what may be important to consider when designing technology to support it. We present a qualitative exploratory study using the `Elicitation Interview' method to obtain a first-person verbal description of experiential processes. We conducted an initial study with 6 experienced professionals from the fashion and textiles area. The analysis revealed that there are two types of touch behaviour in experiencing textiles, active and passive, which happen through `Active hand', `Passive body' and `Active tool-hand'. They can occur in any order, and with different degrees of importance and frequency in the 3 tactile-based phases of the textile selection process - `Situate', `Simulate' and `Stimulate' - and the interaction has different modes in each. We discuss these themes to inform the design of technology for affective touch in the textile field, but also to explore a methodology to uncover the complexity of affective touch and its various purposes.

An ever more sedentary lifestyle is a serious problem in our society. Enhancing people's exercise adherence through technology remains an important research challenge. We propose a novel approach for a system supporting walking that draws from basic findings in neuroscience research. Our shoe-based prototype senses a person's footsteps and alters in real-time the frequency spectra of the sound they produce while walking. The resulting sounds are consistent with those produced by either a lighter or heavier body. Our user study showed that modified walking sounds change one's own perceived body weight and lead to a related gait pattern. In particular, augmenting the high frequencies of the sound leads to the perception of having a thinner body and enhances the motivation for physical activity inducing a more dynamic swing and a shorter heel strike. We here discuss the opportunities and the questions our findings open.

Activity trackers are increasingly popular, but they have high levels of abandonment and little evidence exists to suggest why this is. This paper explores barriers to engagement with activity trackers. We extend previous research by not only characterising the barriers users experienced, such as tracking accuracy and device aesthetics, but also by reporting the workarounds they created. We discuss implications for the design of activity tracking systems by reflecting on these workarounds, the potential for activity tracker design to help overcome existing barriers, and how customisation could play a role.

Touch is a primary nonverbal communication channel used to communicate emotions or other social messages. Despite its importance, this channel is still very little explored in the affective computing field, as much more focus has been placed on visual and aural channels. In this paper, we investigate the possibility to automatically discriminate between different social touch types. We propose five distinct feature sets for describing touch behaviours captured by a grid of pressure sensors. These features are then combined together by using the Random Forest and Boosting methods for categorizing the touch gesture type. The proposed methods were evaluated on both the HAART (7 gesture types over different surfaces) and the CoST (14 gesture types over the same surface) datasets made available by the Social Touch Gesture Challenge 2015. Well above chance level performances were achieved with a 67% accuracy for the HAART and 59% for the CoST testing datasets respectively.

Despite its importance in social interactions, laughter remains little studied in affective computing. Intelligent virtual agents are often blind to users’ laughter and unable to produce convincing laughter themselves. Respiratory, auditory, and facial laughter signals have been investigated but laughter-related body movements have received less attention. The aim of this study is threefold. First, to probe human laughter perception by analyzing patterns of categorisations of natural laughter animated on a minimal avatar. Results reveal that a low dimensional space can describe perception of laughter “types”. Second, to investigate observers’ perception of laughter (hilarious, social, awkward, fake, and non-laughter) based on animated avatars generated from natural and acted motion-capture data. Significant differences in torso and limb movements are found between animations perceived as laughter and those perceived as non-laughter. Hilarious laughter also differs from social laughter. Different body movement features were indicative of laughter in sitting and standing avatar postures. Third, to investigate automatic recognition of laughter to the same level of certainty as observers’ perceptions. Results show recognition rates of the Random Forest model approach human rating levels. Classification comparisons and feature importance analyses indicate an improvement in recognition of social laughter when localized features and nonlinear models are used.

Physical activity is important for improving quality of life in people with chronic pain. However, actual or anticipated pain exacerbation, and lack of confidence when doing physical activity, make it difficult to maintain and build towards long-term activity goals. Research guiding the design of interactive technology to motivate and support physical activity in people with chronic pain is lacking. We conducted studies with: (1) people with chronic pain, to understand how they maintained and increased physical activity in daily life and what factors deterred them; and (2) pain-specialist physiotherapists, to understand how they supported people with chronic pain. Building on this understanding, we investigated the use of auditory feedback to address some of the psychological barriers and needs identified and to increase self-efficacy, motivation and confidence in physical activity. We conclude by discussing further design opportunities based on the overall findings.

Physical activity is essential in chronic pain rehabilitation. However, anxiety due to pain or a perceived exacerbation of pain causes people to guard against beneficial exercise. Interactive rehabiliation technology sensitive to such behaviour could provide feedback to overcome such psychological barriers. To this end, we developed a Support Vector Machine framework with the feature level fusion of body motion and muscle activity descriptors to discriminate three levels of pain (none, low and high). All subjects underwent a forward reaching exercise which is typically feared among people with chronic back pain. The levels of pain were categorized from control subjects (no pain) and thresholded self reported levels from people with chronic pain. Salient features were identified using a backward feature selection process. Using feature sets from each modality separately led to high pain classification F1 scores of 0.63 and 0.69 for movement and muscle activity respectively. However using a combined bimodal feature set this increased to F1 = 0.8.

Naturalistic affective expressions change at a rate much slower than the typical rate at which video or audio is recorded. This increases the probability that consecutive recorded instants of expressions represent the same affective content. In this paper, we exploit such a relationship to improve the recognition performance of continuous naturalistic affective expressions. Using datasets of naturalistic affective expressions (AVEC 2011 audio and video dataset, PAINFUL video dataset) continuously labeled over time and over different dimensions, we analyze the transitions between levels of those dimensions (e.g., transitions in pain intensity level). We use an information theory approach to show that the transitions occur very slowly and hence suggest modeling them as first-order Markov models. The dimension levels are considered to be the hidden states in the Hidden Markov Model (HMM) framework. Their discrete transition and emission matrices are trained by using the labels provided with the training set. The recognition problem is converted into a best path-finding problem to obtain the best hidden states sequence in HMMs. This is a key difference from previous use of HMMs as classifiers. Modeling of the transitions between dimension levels is integrated in a multistage approach, where the first level performs a mapping between the affective expression features and a soft decision value (e.g., an affective dimension level), and further classification stages are modeled as HMMs that refine that mapping by taking into account the temporal relationships between the output decision labels. The experimental results for each of the unimodal datasets show overall performance to be significantly above that of a standard classification system that does not take into account temporal relationships. In particular, the results on the AVEC 2011 audio dataset outperform all other systems presented at the international competition.

Despite the importance of laughter in social interactions it remains little studied in affective computing. Respiratory, auditory, and facial laughter signals have been investigated but laughter-related body movements have received almost no attention. The aim of this study is twofold: first an investigation into observers' perception of laughter states (hilarious, social, awkward, fake, and non-laughter) based on body movements alone, through their categorization of avatars animated with natural and acted motion capture data. Significant differences in torso and limb movements were found between animations perceived as containing laughter and those perceived as nonlaughter. Hilarious laughter also differed from social laughter in the amount of bending of the spine, the amount of shoulder rotation and the amount of hand movement. The body movement features indicative of laughter differed between sitting and standing avatar postures. Based on the positive findings in this perceptual study, the second aim is to investigate the possibility of automatically predicting the distributions of observer's ratings for the laughter states. The findings show that the automated laughter recognition rates approach human rating levels, with the Random Forest method yielding the best performance.

The main aim of this first workshop on Mediated Touch and Affect (MeTA) is to bring together researchers from diverse communities, such as affective computing, hap tics, augmented reality, communication, design, psychology, human-robot interaction, and telepresence. The goal is to discuss the current state of research on mediated touch and affect and to formulate a research agenda for future directions in research on aspects of the touch-technology-affect triangle.

Laughter is a ubiquitous social signal in human interactions yet it remains understudied from a scientific point of view. The need to understand laughter and its role in human interactions has become more pressing as the ability to create conversational agents capable of interacting with humans has come closer to a reality. This paper reports on three aspects of the human perception of laughter when context has been removed and only the body information from the laughter episode remains. We report on ability to categorise the laugh type and the sex of the laugher, the relationship between personality factors with laughter categorisation and perception, and finally the importance of intensity in the perception and categorisation of laughter.

Handling textiles is not only a semantic experience, but also an emotional one. Whilst handling a textile is crucial for its appreciation and understanding, this channel is still little explored in the digital realm, where focus has been given to the haptic feedback aspect of handling. In this paper, we discuss the importance of touch behaviour in interactive digital handling to allow people to explore, emotionally engage with and understand textile properties. We build on our findings from previous studies, where we investigated how people handle fabrics in real-life situations and more generally how their touch behaviour may affect the experience, relating it to literature from textile, HCI, embodied cognition and embodied affect to discuss how current technology should develop to provide a more realistic touch experience. Additionally, we consider how crowd sourcing of the textile experience could be extended by taking into account non-verbal expressions of textile-handling experience. We show that further knowledge is needed to design interactive technology that supports active and unconstrained touch, as well as the affective aspects of experience.

Not only facial expressions but also body gestures and postures play an important role in non-verbal communication. Whilst evidence shows that affective standing postures account for at least four affective dimensions - arousal, valence, potency and avoidance - it is not clear if the same is true for sitting postures. In addition, whilst there is a large body of work investigating cross-cultural differences in the perception of facial expressions, there is very little work on the cross-cultural perception of body expressions. We investigated the minimal universality and cross-cultural difference in the perception of sitting postures and the body descriptors that guide these perceptual processes. Japanese body postures were collected and measured through conventional sensors. Japanese and British observers were recruited for the cross-cultural study. The results show that, for Japanese observers, the three dimensions of arousal, valence and dominance were necessary to account for most of the variance in the perception of the set of affective postures whilst, for British observers, only valence and arousal were necessary. An analysis of the body descriptors highlighted a few differences in the way these were associated with affective dimensions. It also showed that, for Japanese observers, the rating of dominance was modulated by the position of the legs, arms and trunk.

Current interactive media presentations of textiles provide an impoverished communication of their 'textile hand', that is their weight, drape, how they feel to touch. These are complex properties experienced through the visual, tactile, auditory and proprioceptive senses and are currently lost when textile materials are presented in interactive video. This paper offers a new perspective from which the production of multi-touch interactive video representations of the tactile qualities of materials is considered. Through an understanding of hand properties of textiles and how people inherently touch and handle them, we are able to develop methods to animate and bring these properties alive using design methods. Observational studies were conducted, noting gestures consumers used to evaluate textile hand. Replicating the appropriate textile deformations for these gestures in interactive video was explored as a design problem. The resulting digital textile swatches and their interactive behavior were then evaluated for their ability to communicate tactile qualities similar to those of the real textiles.

Calibration is a necessary step in many sensor-based ubicomp applications to prepare a system for operation. Particularly when dealing with sensors for movement-based interaction calibration is required to individualize the system to the person's body. However, calibration is often viewed as a tedious necessity of a purely technical nature. In this paper we argue that calibration can be used as a valuable and informative step for users molding a technology for their own use. We explain this through two case studies that use body sensing technologies to teach physical skills. Our studies show that calibration can be used by teachers and pupils to set goals. We argue that demystifying calibration and designing to expose the intentions of the technology and its functioning can be beneficial for users, allowing them to shape technology to be in tune with their bodies rather than changing their body to fit the technology.

Many real world datasets occur or can be arranged into multi-modal structures. With such datasets, the tasks to be learnt can be referenced by multiple indices. Current multitask learning frameworks are not designed to account for the preservation of this information. We propose the use of multilinear algebra as a natural way to model such a set of related tasks. We present two learning methods; one is an adapted convex relaxation method used in the context of tensor completion. The second method is based on the Tucker decomposition and on alternating minimization. Experiments on synthetic and real data indicate that the multilinear approaches provide a significant improvement over other multitask learning methods. Overall our second approach yields the best performance in all datasets.

The introduction of full-body controllers has made computer games more accessible and promises to provide a more natural and engaging experience to players. However, the relationship between body movement and game engagement is not yet well understood. In this article, I consider how body movement affects the player's experience during game play. I start by presenting a taxonomy of body movements observed during game play. These are framed in the context of a body of previously published research that is then embedded into a novel model of engagement. This model describes the relationship between the taxonomy of movement and the type of engagement that each class of movement facilitates. I discuss the factors that may inhibit or enhance such relationship. Finally, I conclude by considering how the proposed model could lead to a more systematic and effective use of body movement for enhancing game experience.

Thanks to the decreasing cost of whole-body sensing technology and its increasing reliability, there is an increasing interest in, and understanding of, the role played by body expressions as a powerful affective communication channel. The aim of this survey is to review the literature on affective body expression perception and recognition. One issue is whether there are universal aspects to affect expression perception and recognition models or if they are affected by human factors such as culture. Next, we discuss the difference between form and movement information as studies have shown that they are governed by separate pathways in the brain. We also review psychological studies that have investigated bodily configurations to evaluate if specific features can be identified that contribute to the recognition of specific affective states. The survey then turns to automatic affect recognition systems using body expressions as at least one input modality. The survey ends by raising open questions on data collecting, labeling, modeling, and setting benchmarks for comparing automatic recognition systems.

We describe the insights and challenges offered by researcher participation in in-the-wild studies through the comparison of two prototype evaluations with varying levels of researcher participation. By reflecting on these studies we expose different facets of the researcher's role when interacting with participants in in-the-wild studies. We also demonstrate the value of researcher participation in contributing to the way a researcher understands participant responses: aiding rapport, promoting empathy and stimulating the researcher to reflect on their own assumptions.

The emergence of full-body computer games raises an interesting question: Can body movement be used to measure the aesthetic experience of players? In this paper, we aim to take a first step toward answering this question. Such a question emerges from the fact that various studies have shown the dual role of body movement, i.e., a window on people's emotional and mental states as well as a means to affect people's cognitive and affective processes. In this paper, first, we investigate the possibility of automatically recognizing the emotional expressions conveyed by the player's body movement in a Nintendo sport game. Our results showed that our automatic recognition system achieved recognition rates comparable to human observers' benchmarks. Second, by taking a pragmatist definition of aesthetic experience into account, we argue that the tracked body expressions do not only express what the player may be feeling. Given their modulating role on cognition and affect, these body expressions also let the player actively construct and assign affective meanings to the unfolding of the game. We argue that the player's variety of emotional bodily expressions constitutes the emotional rhythmic dynamic of aesthetic experience and, as such, they provide a measure of its distinctive quality.

The increasing number of people playing games on touch-screen mobile phones raises the question of whether touch behaviors reflect players’ emotional states. This prospect would not only be a valuable evaluation indicator for game designers, but also for real-time personalization of the game experience. Psychology studies on acted touch behavior show the existence of discriminative affective profiles. In this article, finger-stroke features during gameplay on an iPod were extracted and their discriminative power analyzed. Machine learning algorithms were used to build systems for automatically discriminating between four emotional states (Excited, Relaxed, Frustrated, Bored), two levels of arousal and two levels of valence. Accuracy reached between 69% and 77% for the four emotional states, and higher results (~89%) were obtained for discriminating between two levels of arousal and two levels of valence. We conclude by discussing the factors relevant to the generalization of the results to applications other than games.

A novel emotion recognition system has been proposed for classifying facial expression in videos. Firstly, two types of basic facial appearance descriptors were extracted. The first type of descriptor, called Motion History Histogram (MHH), was used to detect temporal changes of each pixels of the face. The second type of descriptor, called Histogram of Local Binary Patterns (LBP), was applied to each frame of the video and was used to capture local textural patterns. Secondly, based on these two basic types of descriptors, two new dynamic facial expression features called MHH_EOH and LBP MCF were proposed. These two features incorporate both dynamic and local information. Finally, the Two View SVK_2K classifier was built to integrate these two dynamic features in an efficient way. The experimental results showed that this method outperformed the baseline results set by the FERA'11 challenge.

The conveyance and recognition of affect and emotion partially determine how people interact with others and how they carry out and perform in their day-to-day activities. Hence, it is becoming necessary to endow technology with the ability to recognize users' affective states to increase the technologies' effectiveness. This paper makes three contributions to this research area. First, we demonstrate recognition models that automatically recognize affective states and affective dimensions from non-acted body postures instead of acted postures. The scenario selected for the training and testing of the automatic recognition models is a body-movement-based video game. Second, when attributing affective labels and dimension levels to the postures represented as faceless avatars, the level of agreement for observers was above chance level. Finally, with the use of the labels and affective dimension levels assigned by the observers as ground truth and the observers' level of agreement as base rate, automatic recognition models grounded on low-level posture descriptions were built and tested for their ability to generalize to new observers and postures using random repeated subsampling validation. The automatic recognition models achieve recognition percentages comparable to the human base rates as hypothesized.

Multiple autonomous Unmanned Aerial Vehicles (UAVs) can be used to complement human teams. This paper presents the results of an exploratory study to investigate gesture/speech interfaces for interaction with robots in a situated manner and the development of three iterations of a prototype command set. A command set was compiled from observing users interacting with a simulated interface in a virtual reality environment. We discovered that users find this type of interface intuitive and their commands tend to naturally group into both `High-Level' and `Low-Level' instructions. However, as the robots moved further away, the loss of depth perception and direct feedback was inimical to the interaction. In a second experiment we found that using simple heads up display elements could mitigate these issues.

The recent development of controllers designed around natural body movements has altered the nature of gaming and contributed towards it being marketed as a more social activity. The study reported here compares the use of Donkey Konga bongos with a standard controller to examine how affording motion through an input device affects social interaction. Levels of engagement with the game were also measured to explore whether increases in social behaviour in the 'real world' would result in reduced involvement with the 'game world'. Social interaction was significantly higher when the bongos were used, but this did not detract from engagement. Instead, engagement was also found to increase when body movement was afforded.

Conveyance and recognition of human emotion and affective expression is influenced by many factors, including culture. Within the user modeling field, it has become increasingly necessary to understand the role affect can play in personalizing interactive interfaces using embodied animated agents. However, little research within the computer science field aims at understanding cultural differences within this vein. Therefore, we conducted a study to evaluate if differences exist in the way various cultures perceive emotion from body posture. We used static posture images of affectively expressive avatars to conduct recognition experiments with subjects from three cultures. After analyzing the subjects' judgments using multivariate analysis, we grounded the identified differences into a set of low-level posture features. We then used Mixture Discriminant Analysis (MDA) and an unsupervised expectation maximization (EM) model to build separate cultural models for affective posture recognition. Our results could prove useful to aide designers in creating more effective affective avatars.

Today, creating systems that are capable of interacting naturally and efficiently with humans on many levels is essential. One step toward achieving this is the recognition of emotion from whole body postures of human partners. Currently, little research in this area exists in computer science. Therefore, our aim is to identify and measure the saliency of posture features that play a role in affective expression. As a case-study, we collected affective gestures from human subjects using a motion capture system. We first described these gestures with spatial features. Through standard statistical techniques, we verified that there was a statistically significant correlation between the emotion intended by the acting subjects, and the emotion perceived by the observers. We examined the use of discriminant analysis to measure the saliency of the proposed set of posture features in discriminating between 4 basic emotions: angry, fear, happy, and sad. Our results show that the set of features discriminates well between emotions, and also provides evidence about the strong overlap between descriptors in both acting and observing activities.

Connection Science, Vol. 15, No. 4, December 2003, 259–269 Page 259, line 6, reads: [email protected] It should read: [email protected] Taylor and Francis Ltd apologises for this error and for...

Abstract One of the challenging issues in affective computing is to give a machine the ability to recognize the mood of a person. Efforts in that direction have mainly focused on facial and oral cues. Gestures have been recently considered as well, but with less success. Our aim is to fill this gap by identifying and measuring the saliency of posture features that play a role in affective expression. As a case study, we collected affective gestures from human subjects using a motion capture system. We first described these gestures with spatial features, as suggested in studies on dance. Through standard statistical techniques, we verified that there was a statistically significant correlation between the emotion intended by the acting subjects, and the emotion perceived by the observers. We used Discriminant Analysis to build affective posture predictive models and to measure the saliency of the proposed set of posture features in discriminating between 4 basic emotional states: angry, fear, happy, and sad. An information theoretic characterization of the models shows that the set of features discriminates well between emotions, and also that the models built over‐perform the human observers. Copyright © 2004 John Wiley & Sons, Ltd.

Nowadays, a major activity on the Internet is the retrieval and browsing of multimedia information. Yet, today 's search engines are not really up to the task. Users often have to query various search engines and browse many Web sites before finding a satisfactory answer. Once users find such an answer, presentation of the results is very rigid, i.e., not tailored to the users' needs, and it does not enable users to manipulate the data. Thus, this retrieval activity often becomes tedious. To improve user-information interaction, a major issue is to give search-engines the ability to access data semantics. We propose a framework to address the issue. Our framework combines database and multimedia data mining to endow Web-based applications with the ability to let users manipulate the data at different levels of interest. As an experimental testbed, we implemented a holiday planner that supports users in their search for a hotel. By using database technology, Web-based multimedia interaction becomes possible, offering as a side effect more feasible and reliable feedback.

Studies on emotion are currently receiving a lot of attention. The importance of emotion in the development and support of intelligent and social behaviour has been highlighted by studies in psychology and neurology. Hence, the recognition of affective states has also become a critical feature in robot social development, with robots assumed to take on a role as social companion. In this paper, we address the issue of endowing robots with the ability to learn incrementally to recognize the affective state of their human partner by interpreting their gestural cues. We propose a model that can self-organize postural features into affective categories, and use contextual feedback from the partner to drive the learning process.

We explore interaction as a basis for human-computer comprehension. Perceptual experience is organised through categories establishing the kind of distinctions imposable on perceived phenomena. An interactive tool exploiting a similar categorisation in graphics is integrated into a system for subjective retrieval of images, so that the user interacts with active regions supporting the same type of distinction.

We suggest that four points should be considered when developing new information retrieval systems: a) taking information in its entirety (interpretation), b) managing multimedia information in an integrated way, c) reusing already available tools and d) tailoring information to single users. We propose a framework for endowing a software agent with the capacity to personalize itself to its users through interaction, and to perform tasks that involve subjective factors. We describe K-DIME, a software prototype that can retrieve material from the Web based on both objective and subjective features of content. Being capable of bootstrapping a new user model from a model of a user with similar profile, K-DIME significantly reduces the workload generally associated with the online learning phase. Through continuous adaptation driven by specific patterns of interaction with the user, K-DIME can cope with the intrinsic variability of subjective impressions. We discuss K-DIME's performance in a scenario in which users retrieve pictures for greeting cards according to a given subjective impression that they wish to convey.

Recently, in designing computing systems, much focus has been given to support the user's subjectivity (Kansei in Japanese), by providing the system with a mostly explicit user model. We argue against this explicit nature of the model since there is evidence in neuroscience that the human brain does not have monolithic control and static internal models. We suggest that the user model should rather emerge from a prolonged interaction between the user and the system. We describe our interactive visual environment dedicated to image retrieval based on subjective parameters. It is endowed with a learning agent and an active interface allowing multi-model interaction between the user and the system. This interaction takes the form of symbols, examples or externalization of internal processes.

The paper addresses the problem of controlling situated image understanding processes. Two complementary control styles are considered and applied cooperatively, a deliberative one and a reactive one. The role of deliberative control is to account for the unpredictability of situations, by dynamically determining which strategies to pursue, based on the results obtained so far and more generally on the state of the understanding process. The role of reactive control is to account for the variability of local properties of the image by tuning operations to subimages, each one being homogeneous with respect to a given operation. A variable organization of agents is studied to face this variability. The two control modes are integrated into a unified formalism describing segmentation and interpretation activities. A feedback from high level interpretation tasks to low level segmentation tasks thus becomes possible and is exploited to recover wrong segmentations. Preliminary results in the field of liver biopsy image understanding are shown to demonstrate the potential of the approach.

Two complementary control styles, deliberative and reactive types, are applied to account for unpredictability of situations and variability of local properties in image interpretation processes. Deliberative control is based on attributed grammars and local variability is managed by a variable organisation of agents. Feedback from interpretation to segmentation becomes possible and is exploited to recover wrong segmentation. Experimental results illustrate the potential of the approach.