FFAME (Filtering Familiar Audio for Movement Exploration) is a novel sonification framework aiming to facilitate movement in individuals with chronic back pain. Our personalised, music-based approach contrasts and extends prior work with predetermined tonal sonification. FFAME progressively filters selected music based on angles of the trunk. Through a qualitative analysis of reported experience of 15 participants with chronic pain and 5 physiotherapists, we identify how sonification parameters and musical characteristics affect movement and meaning-making. Music-based movement sonification proved impactful across multiple dimensions: (1) encouraging movement, (2) escaping pain-related rumination, (3) externalizing pain experiences, and (4) scaffolding physical activities. Drawing on enactivism and related philosophies, the study highlights how the semantic indeterminacy of music, combined with real-time movement sonification, created a rich, open-ended environment that supported user agency and exploration. Sonification for pain management can be creative and expressive, enabling people with pain to extend challenging movements and build movement confidence.

Chronic pain is a prevalent condition where fear of movement and pain interfere with everyday functioning. Yet, there is no open body movement dataset for people with chronic pain in everyday settings. Our EmoPain@Home dataset addresses this with capture from 18 people with and without chronic pain in their homes, while they performed their routine activities. The data includes labels for pain, worry, and movement confidence continuously recorded for activity instances for the people with chronic pain. We explored baseline two-level pain detection based on this dataset and obtained 0.62 mean F1 score. However, extension of the dataset led to deterioration in performance confirming high variability in pain expressions for real world settings. We investigated baseline activity recognition for this setting as a first step in exploring the use of the activity label as contextual information for improving pain level classification performance. We obtained mean F1 score of 0.43 for 9 activity types, highlighting its feasibility. Further exploration, however, showed that data from healthy people cannot be easily leveraged for improving performance because worry and low confidence alter activity strategies for people with chronic pain. Our dataset and findings lay critical groundwork for automatic assessment of pain experience and behaviour in the wild.

Self-supervised learning has shown value for uncovering informative movement features for human activity recognition. However, there has been minimal exploration of this approach for affect recognition where availability of large labelled datasets is particularly limited. In this paper, we propose a P-STEMR (Parallel Space-Time Encoding Movement Representation) architecture with the aim of addressing this gap and specifically leveraging the higher availability of human activity recognition datasets for pain-level classification. We evaluated and analyzed the architecture using three different datasets across four sets of experiments. We found statistically significant increase in average F1 score to 0.84 for pain level classification with two classes based on the architecture compared with the use of hand-crafted features. This suggests that it is capable of learning movement representations and transferring these from activity recognition based on data captured in lab settings to classification of pain levels with messier real-world data. We further found that the efficacy of transfer between datasets can be undermined by dissimilarities in population groups due to impairments that affect movement behaviour and in motion primitives (e.g. rotation versus flexion). Future work should investigate how the effect of these differences could be minimized so that data from healthy people can be more valuable for transfer learning.

Chronic pain is a prevalent condition that affects everyday life of people around the world. Protective behaviors (strategies that are naturally but unhelpfully adopted by people with chronic pain to cope with fear of pain in executing harmless everyday movements) can lead to further disability over time if not recognized and addressed appropriately. In this paper, we build on previous work on unimodal, activity-independent, time-continuous protective behavior detection (PBD) by focusing on the fusion of muscle activity and body movement modalities for characterizing both protective behavior and its physical activity context. We explore different fusion strategies based on consideration of the manner in which protective behavior influences muscle activity and overt body movement as well as the relationship between the two modalities. We evaluate the various strategies on the multimodal EmoPain dataset containing data from people with and without chronic pain engaged in physical activities that reflect everyday challenges for those with chronic pain. Our results show that a central (model-level) fusion approach leads to better PBD performance than input- and decision-level fusions, or unimodal approaches. We also show that additional use of attention mechanism, typifying shifts in attention characteristic of protective behavior, further improves the sensitivity of the model, i.e. detection of the positive class (which is the minority class). We analyze these results and suggest that fusion in modelling a motor condition should consider how emotional responses (fear of movement and pain in this case) triggered by a condition affect each of the given modalities and hence their contributions to the modelling task.

While there is growing interest in developing tech-nology to support pain assessment, pain-related self-management, and healthcare personalisation, there are currently no datasets on nonverbal pain behaviour in the context of functional activities. To address this gap, we introduce the EmoPain(at)Home dataset which consists of motion capture data and self-reported pain, worry, and confidence intensities captured from people with chronic pain. The data were recorded during self-selected functional activities in the home, e.g. vacuuming. We include analysis of the dataset as well as baseline classification of pain levels with average F1 score of 0.61 for two classes. We additionally discuss inclusivity considerations for capture of datasets in naturalistic settings, based on lessons learnt within our study.

Computational systems that process multiple affective states may benefit from explicitly considering the interaction between the states to enhance their recognition performance. This work proposes the combination of a multi-label classifier, Circular Classifier Chain (CCC), with a multimodal classifier, Fusion using a Semi-Naive Bayesian classifier (FSNBC), to include explicitly the dependencies between multiple affective states during the automatic recognition process. This combination of classifiers is applied to a virtual rehabilitation context of post-stroke patients. We collected data from post-stroke patients, which include finger pressure, hand movements, and facial expressions during ten longitudinal sessions. Videos of the sessions were labelled by clinicians to recognize four states: tiredness, anxiety, pain, and engagement. Each state was modelled by the FSNBC receiving the information of finger pressure, hand movements, and facial expressions. The four FSNBCs were linked in the CCC to exploit the dependency relationships between the states. The convergence of CCC was reached by 5 iterations at most for all the patients. Results (ROC AUC) of CCC with the FSNBC are over$0.940 \pm 0.045$($mean \pm std.\;deviation$) for the four states. Relationships of mutual exclusion between engagement and all the other states and co-occurrences between pain and anxiety were detected and discussed.

In chronic pain rehabilitation, physiotherapists adapt physical activity to patients’ performance based on their expression of protective behavior, gradually exposing them to feared but harmless and essential everyday activities. As rehabilitation moves outside the clinic, technology should automatically detect such behavior to provide similar support. Previous works have shown the feasibility of automatic protective behavior detection (PBD) within a specific activity. In this article, we investigate the use of deep learning for PBD across activity types, using wearable motion capture and surface electromyography data collected from healthy participants and people with chronic pain. We approach the problem by continuously detecting protective behavior within an activity rather than estimating its overall presence. The best performance reaches mean F1 score of 0.82 with leave-one-subject-out cross validation. When protective behavior is modeled per activity type, performance achieves a mean F1 score of 0.77 for bend-down, 0.81 for one-leg-stand, 0.72 for sit-to-stand, 0.83 for stand-to-sit, and 0.67 for reach-forward. This performance reaches excellent level of agreement with the average experts’ rating performance suggesting potential for personalized chronic pain management at home. We analyze various parameters characterizing our approach to understand how the results could generalize to other PBD datasets and different levels of ground truth granularity.

The EmoPain 2020 Challenge is the first international competition aimed at creating a uniform platform for the comparison of multi-modal machine learning and multimedia processing methods of chronic pain assessment from human expressive behaviour, and also the identification of pain-related behaviours. The objective of the challenge is to promote research in the development of assistive technologies that help improve the quality of life for people with chronic pain via real-time monitoring and feedback to help manage their condition and remain physically active. The challenge also aims to encourage the use of the relatively underutilised, albeit vital bodily expression signals for automatic pain and pain-related emotion recognition. This paper presents a description of the challenge, competition guidelines, bench-marking dataset, and the baseline systems' architecture and performance on the Challenge's three sub-tasks: pain estimation from facial expressions, pain recognition from multimodal movement, and protective movement behaviour detection.

Clinicians tailor intervention in chronic pain rehabilitation to movement related self-efficacy (MRSE). This motivates us to investigate automatic MRSE estimation in this context towards the development of technology that is able to provide appropriate support in the absence of a clinician. We first explored clinical observer estimation, which showed that body movement behaviours, rather than facial expressions or engagement behaviours, were more pertinent to MRSE estimation during physical activity instances. Based on our findings, we built a system that estimates MRSE from bodily expressions and bodily muscle activity captured using wearable sensors. Our results (F1 scores of 0.95 and 0.78 in two physical exercise types) provide evidence of the feasibility of automatic MRSE estimation to support chronic pain physical rehabilitation. We further explored automatic estimation of MRSE with a reduced set of low-cost sensors to investigate the possibility of embedding such capabilities in ubiquitous wearable devices to support functional activity. Our evaluation for both exercise and functional activity resulted in F1 score of 0.79. This result suggests the possibility of (and calls for more studies on) MRSE estimation during everyday functioning in ubiquitous settings. We provide a discussion of the implication of our findings for relevant areas.

The automatic recognition of multiple affective states can be enhanced if the underpinning computational models explicitly consider the interactions between the states. This work proposes a computational model that incorporates the dependencies between four states (tiredness, anxiety, pain, and engagement)known to appear in virtual rehabilitation sessions of post-stroke patients, to improve the automatic recognition of the patients' states. A dataset of five stroke patients which includes their fingers' pressure (PRE), hand movements (MOV)and facial expressions (FAE)during ten sessions of virtual rehabilitation was used. Our computational proposal uses the Semi-Naive Bayesian classifier (SNBC)as base classifier in a multiresolution approach to create a multimodal model with the three sensors (PRE, MOV, and FAE)with late fusion using SNBC (FSNB classifier). There is a FSNB classifier for each state, and they are linked in a circular classifier chain (CCC)to exploit the dependency relationships between the states. Results of CCC are over 90% of ROC AUC for the four states. Relationships of mutual exclusion between engagement and all the other states and some co-occurrences between pain and anxiety for the five patients were detected. Virtual rehabilitation platforms that incorporate the automatic recognition of multiple patient's states could leverage intelligent and empathic interactions to promote adherence to rehabilitation exercises.

In chronic pain physical rehabilitation, physiotherapists adapt exercise sessions according to the movement behavior of patients. As rehabilitation moves beyond clinical sessions, technology is needed to similarly assess movement behaviors and provide such personalized support. In this paper, as a first step, we investigate automatic detection of protective behavior (movement behavior due to pain-related fear or pain) based on wearable motion capture and electromyography sensor data. We investigate two recurrent networks (RNN) referred to as stacked-LSTM and dual-stream LSTM, which we compare with related deep learning (DL) architectures. We further explore data augmentation techniques and additionally analyze the impact of segmentation window lengths on detection performance. The leading performance of 0.815 mean F1 score achieved by stacked-LSTM provides important grounding for the development of wearable technology to support chronic pain physical rehabilitation during daily activities.

Although clinical best practice suggests that affect awareness could enable more effective technological support for physical rehabilitation through personalisation to psychological needs, designers need to consider what affective states matter, and how they should be tracked and addressed. In this article, we set the standard by analysing how the major affective factors in chronic pain (pain, fear/anxiety, and low/depressed mood) interfere with everyday physical functioning. Further, based on discussion of the modality that should be used to track these states to enable technology to address them, we investigated the possibility of using movement behaviour to automatically detect the states. Using two body movement datasets on people with chronic pain, we show that movement behaviour enables very good discrimination between two emotional distress levels (F1=0.86), and three pain levels (F1=0.9). Performance remained high (F1=0.78 for two pain levels) with a reduced set of movement sensors. Finally, in an overall discussion, we suggest how technology-provided encouragement and awareness can be personalised given the capability to automatically monitor the relevant states, towards addressing the barriers that they pose. In addition, we highlight movement behaviour features to be tracked to provide technology with information necessary for such personalisation.

While most rehabilitation technologies target situated exercise sessions and associated performance metrics, physiotherapists recommend physical activities that are integrated with everyday functioning. We conducted a 1-2 week home study to explore how people with chronic pain use wearable technology that senses and sonifies movement (i.e., movement mapped to sound in real-time) to do functional activity (e.g., loading the dishwasher). Our results show that real-time movement sonification led to an increased sense of control during challenging everyday tasks. Sonification calibrated to functional activity facilitated application of pain management techniques such as pacing. When calibrated to individual needs, sonification enabled serendipitous discovery of physical capabilities otherwise obscured by a focus on pain or a dysfunctional proprioceptive system. A physiotherapist was invited to comment on the implications of our findings. We conclude by discussing opportunities provided by wearable sensing technology to enable better functioning, the ultimate goal of physical rehabilitation.

In self-directed chronic pain physical rehabilitation it is important that the individual can progress as physical capabilities and confidence grow. However, people with chronic pain often struggle to pass what they have identified as safe boundaries. At the same time, over-activity due to the desire to progress fast or function more normally, may lead to setbacks. We investigate how musically-informed movement sonification can be used as an implicit mechanism to both avoid overdoing and facilitate progress during stretching exercises. We sonify an end target-point in a stretch exercise, using a stable sound (i.e., where the sonification is musically resolved) to encourage movements ending and an unstable sound (i.e., musically unresolved) to encourage continuation. Results on healthy participants show that instability leads to progression further beyond the target-point while stability leads to a smoother stop beyond this point. We conclude discussing how these findings should generalize to the CP population.

Chronic (persistent) pain (CP) affects 1 in 10 adults; clinical resources are insufficient, and anxiety about activity restricts lives. Technological aids monitor activity but lack necessary psychological support. This article proposes a new sonification framework, Go-with-the-Flow, informed by physiotherapists and people with CP. The framework proposes articulation of user-defined sonified exercise spaces (SESs) tailored to psychological needs and physical capabilities that enhance body and movement awareness to rebuild confidence in physical activity. A smartphone-based wearable device and a Kinect-based device were designed based on the framework to track movement and breathing and sonify them during physical activity. In control studies conducted to evaluate the sonification strategies, people with CP reported increased performance, motivation, awareness of movement, and relaxation with sound feedback. Home studies, a focus group, and a survey of CP patients conducted at the end of a hospital pain management session provided an in-depth understanding of how different aspects of the SESs and their calibration can facilitate self-directed rehabilitation and how the wearable version of the device can facilitate transfer of gains from exercise to feared or demanding activities in real life. We conclude by discussing the implications of our findings on the design of technology for physical rehabilitation.

Pain-related emotions are a major barrier to effective self rehabilitation in chronic pain. Automated coaching systems capable of detecting these emotions are a potential solution. This paper lays the foundation for the development of such systems by making three contributions. First, through literature reviews, an overview of how pain is expressed in chronic pain and the motivation for detecting it in physical rehabilitation is provided. Second, a fully labelled multimodal dataset (named `EmoPain') containing high resolution multiple-view face videos, head mounted and room audio signals, full body 3D motion capture and electromyographic signals from back muscles is supplied. Natural unconstrained pain related facial expressions and body movement behaviours were elicited from people with chronic pain carrying out physical exercises. Both instructed and non-instructed exercises were considered to reflect traditional scenarios of physiotherapist directed therapy and home-based self-directed therapy. Two sets of labels were assigned: level of pain from facial expressions annotated by eight raters and the occurrence of six pain-related body behaviours segmented by four experts. Third, through exploratory experiments grounded in the data, the factors and challenges in the automated recognition of such expressions and behaviour are described, the paper concludes by discussing potential avenues in the context of these findings also highlighting differences for the two exercise scenarios addressed.

People with chronic musculoskeletal pain would benefit from technology that provides run-time personalized feedback and help adjust their physical exercise plan. However, increased pain during physical exercise, or anxiety about anticipated pain increase, may lead to setback and intensified sensitivity to pain. Our study investigates the possibility of detecting pain levels from the quality of body movement during two functional physical exercises. By analyzing recordings of kinematics and muscle activity, our feature optimization algorithms and machine learning techniques can automatically discriminate between people with low level pain and high level pain and control participants while exercising. Best results were obtained from feature set optimization algorithms: 94% and 80% for the full trunk flexion and sit-to-stand movements respectively using Support Vector Machines. As depression can affect pain experience, we included participants' depression scores on a standard questionnaire and this improved discrimination between the control participants and the people with pain when Random Forests were used.

Physical activity is important for improving quality of life in people with chronic pain. However, actual or anticipated pain exacerbation, and lack of confidence when doing physical activity, make it difficult to maintain and build towards long-term activity goals. Research guiding the design of interactive technology to motivate and support physical activity in people with chronic pain is lacking. We conducted studies with: (1) people with chronic pain, to understand how they maintained and increased physical activity in daily life and what factors deterred them; and (2) pain-specialist physiotherapists, to understand how they supported people with chronic pain. Building on this understanding, we investigated the use of auditory feedback to address some of the psychological barriers and needs identified and to increase self-efficacy, motivation and confidence in physical activity. We conclude by discussing further design opportunities based on the overall findings.

Physical activity is essential in chronic pain rehabilitation. However, anxiety due to pain or a perceived exacerbation of pain causes people to guard against beneficial exercise. Interactive rehabiliation technology sensitive to such behaviour could provide feedback to overcome such psychological barriers. To this end, we developed a Support Vector Machine framework with the feature level fusion of body motion and muscle activity descriptors to discriminate three levels of pain (none, low and high). All subjects underwent a forward reaching exercise which is typically feared among people with chronic back pain. The levels of pain were categorized from control subjects (no pain) and thresholded self reported levels from people with chronic pain. Salient features were identified using a backward feature selection process. Using feature sets from each modality separately led to high pain classification F1 scores of 0.63 and 0.69 for movement and muscle activity respectively. However using a combined bimodal feature set this increased to F1 = 0.8.