Learning clothes repair is challenging for novices, who face interconnected procedural and embodied challenges, especially when learning alone. Existing tools fail to provide holistic support as interactive tutors and lack awareness of the embodied interactions of working with textiles. This paper presents a multi-phase study that investigates these challenges and explores the design space for a Human-Touch-Aware conversational agent (RepairBot). We began with an in-depth autoethnography to understand the novice experience, which informed the development of the RepairBot Conversation Framework (RBCF) together with a design implementation of a technology probe. Using the RepairBot prototype together with a Wizard-of-Oz approach to simulate Human-Touch-Awareness, we investigated how a conversational agent could support repair learning in novices as well as engage them with their own clothes-repairing projects. Subsequent lab and in-home studies with novice participants suggested specific conversational and embodied mechanisms that would facilitate novices’ holistic understanding of repair, increase their confidence, and elicit attentive touch and emotional reflection. We bring these mechanisms together in the framework presented in this paper.

Aligning large language models (LLMs) behaviour with human intent is critical for future AI. An important yet often overlooked aspect of this alignment is the perceptual alignment. Perceptual modalities like touch are more multifaceted and nuanced compared to other sensory modalities such as vision. This study investigates how well LLMs can understand and interpret human touch experiences by focusing on their capacity to perceive the tactile qualities of everyday objects. For instance, it assesses whether LLMs can recognize that silk satin is softer and smoother than cotton denim. We developed a “Guess What Textile“ interaction using a custom AI system that enables participants to narrate their touch experiences in the “textile hand” task. Participants were given two textile samples–a target and a reference–to handle. Without seeing them, participants described the differences between them to the LLM. Using these descriptions, the LLM attempted to identify the target textile by assessing similarity within its high-dimensional embedding space, where its perceptual representations are encoded. Our results suggest that a degree of perceptual alignment exists; however, it varies significantly among different textile samples. For example, LLM predictions are well aligned for silk satin, but not for cotton denim. Moreover, participants felt that their textile experiences were not closely matched by the LLM predictions. This study is the first exploration into perceptual alignment around touch using LLM encoders, exemplified through textile hand task. We discuss possible sources of this alignment variance, and how better human-AI perceptual alignment can benefit future everyday tasks. • We address the gap in understanding perception alignment between human touch and AI. • First study on alignment between human touch experiences and LLMs in embeddings. • A novel interactive task probes LLMs’ learned representations for human alignment. • LLMs show perceptual biases, aligning better with certain textiles than others.

Augmented reality displays are becoming more powerful and simultaneously more mobile. Although mobile AR is gaining popularity, it remains difficult to get an insight into users' cognitive load, despite its relevance for many mobile-based tasks. Usually, cognitive load is measured via subjective, task-disruptive self-reports such as NASA TLX. While biosensors such as galvanic skin response, heart rate variability, or pulse have been used to obtain more objective measures, these are highly susceptible to motion-induced noise. More robust techniques like EEG offer higher reliability but are impractical for mobile, real-world use. In this paper, we report on a non-contact multi-sensor approach to assess cognitive load in mobile AR. Our approach combines pupillometry, facial expression tracking, and thermal imaging for respiratory rate analysis. Within the frame of our study, we analysed the aptness of the methods, comparing load assessment for low and high cognitive load tasks under both stationary and mobile conditions. Using an XGBoost classifier, our model achieved 86.11% accuracy for binary cognitive load assessment (low vs. high cognitive load) and 84.24% accuracy for four-way classification (cognitive load$\times$mobility). Feature importance analysis revealed that robust predictors included gaze dynamics (e.g., fixation, pursuit, and saccade durations), pupil diameter metrics (such as FFT band power and variability measures), and facial and respiratory features (including brow lowering and nostril temperature quantiles) for assessing cognitive load in mobile AR.

Speech impairments caused by severe diseases and injuries are serious health problems. The deprivation of communication ability can significantly decrease the quality of daily life. Inner speech is a natural mental activity used by many healthy and disabled people with intact cognitive ability. With the help of Brain-Computer Interfaces (BCIs), imagined speech can be decoded as semantic output or downstream commands for external devices, showing great potential for intuitive neural interface control. As many works have demonstrated promising results with invasive brain recordings, it remains a challenge for non-invasive BCIs to realize reliable speech decoding due to the trade-off between safety and signal quality. In this study, we explored the feasibility and neural mechanism behind a non-invasive brain recording technique based on electroencephalogram (EEG) during speech imagery with 6 English phonemes. We found significant time-frequency representation differences between /b/ and /u:/ and showed feasibility for pair-wise imagined phoneme classification with Filter Bank Common Spatial Pattern. We also demonstrated the EEG neural representation of phonemes in the latent space and how they are separated. Our results suggested that for naïve BCI users, subject-specific phoneme pairs yielded the best performance. Finally, we discussed the challenges and potential optimization directions for future EEG-based speech BCIs.

Human-artificial intelligence (AI) alignment ensures that AI systems align with human goals and behaviors. This paper introduces perceptual alignment as a critical aspect of this alignment, focusing on the concurrence between human judgments and AI evaluations across sensory modalities. We particularly explore how Multimodal Large Language Models (MLLMs), which process both visual and textual data, interpret the tactile qualities of textiles—a significant challenge in online shopping environments. Our research analyzes six vision-based MLLMs to see how they describe the tactile experience of textiles and compares these AI-generated descriptions with human assessments. Through semantic similarity measures and in-person evaluations, we investigate the extent of alignment between human perceptions and AI descriptions. Our findings indicate significant variability in the AI’s ability to interpret different textiles, highlighting both the potential and limitations of current AI models in achieving perceptual alignment. This work contributes to understanding the complexities of aligning AI capabilities with human touch sensory experiences.

Remote photoplethysmography (rPPG) enables non-invasive extraction of blood volume pulse signals through imaging, transforming spatial-temporal data into time series signals. Advances in end-to-end rPPG approaches have focused on this transformation where attention mechanisms are crucial for feature extraction. However, existing methods compute attention disjointly across spatial, temporal, and channel dimensions. Here, we propose the Factorized Self-Attention Module (FSAM), which jointly computes multidimensional attention from voxel embeddings using nonnegative matrix factorization. To demonstrate FSAM's effectiveness, we developed FactorizePhys, an end-to-end 3D-CNN architecture for estimating blood volume pulse signals from raw video frames. Our approach adeptly factorizes voxel embeddings to achieve comprehensive spatial, temporal, and channel attention, enhancing performance of generic signal extraction tasks. Furthermore, we deploy FSAM within an existing 2D-CNN-based rPPG architecture to illustrate its versatility. FSAM and FactorizePhys are thoroughly evaluated against state-of-the-art rPPG methods, each representing different types of architecture and attention mechanism. We perform ablation studies to investigate the architectural decisions and hyperparameters of FSAM. Experiments on four publicly available datasets and intuitive visualization of learned spatial-temporal features substantiate the effectiveness of FSAM and enhanced cross-dataset generalization in estimating rPPG signals, suggesting its broader potential as a multidimensional attention mechanism. The code is accessible at https://github.com/PhysiologicAILab/FactorizePhys.

Touch exploration of fabric is used to evaluate its properties, and it could further be leveraged to understand a consumer’s sensory experience and preference so as to support them in real time to make careful clothing purchase decisions. In this paper, we open up opportunities to explore the use of technology to provide such support with our FabricTouch dataset, i.e., a multimodal dataset of fabric assessment touch gestures. The dataset consists of bilateral forearm movement and muscle activity data captured while 15 people explored 114 different garments in total to evaluate them according to 5 properties (warmth, thickness, smoothness, softness, and flexibility). The dataset further includes subjective ratings of the garments with respect to each property and ratings of pleasure experienced in exploring the garment through touch. We further report baseline work on automatic detection. Our results suggest that it is possible to recognise the type of fabric property that a consumer is exploring based on their touch behaviour. We obtained mean F1 score of 0.61 for unseen garments, for 5 types of fabric property. The results also highlight the possibility of additionally recognizing the consumer’s subjective rating of the fabric when the property being rated is known, mean F1 score of 0.97 for unseen subjects, for 3 rating levels.

Ninety percent of the 1.2 billion people who need assistive technology (AT) do not have access. Information seeking practices directly impact the ability of AT producers, procurers, and providers (AT professionals) to match a user's needs with appropriate AT, yet the AT marketplace is interdisciplinary and fragmented, complicating information seeking. We explored common limitations experienced by AT professionals when searching information to develop solutions for a diversity of users with multi-faceted needs. Through Template Analysis of 22 expert interviews, we find current search engines do not yield the necessary information, or appropriately tailor search results, impacting individuals’ awareness of products and subsequently their availability and the overall effectiveness of AT provision. We present value-based design implications to improve functionality of future AT-information seeking platforms, through incorporating smarter systems to support decision-making and need-matching whilst ensuring ethical standards for disability fairness remain.

As an emerging interaction paradigm, physiological computing is increasingly being used to both measure and feed back information about our internal psychophysiological states. While most applications of physiological computing are designed for individual use, recent research has explored how biofeedback can be socially shared between multiple users to augment human-human communication. Reflecting on the empirical progress in this area of study, this paper presents a systematic review of 64 studies to characterize the interaction contexts and effects of social biofeedback systems. Our findings highlight the importance of physio-temporal and social contextual factors surrounding physiological data sharing as well as how it can promote social-emotional competences on three different levels: intrapersonal, interpersonal, and task-focused. We also present the Social Biofeedback Interactions framework to articulate the current physiological-social interaction space. We use this to frame our discussion of the implications and ethical considerations for future research and design of social biofeedback interfaces.

Comfort is a subjective experience that people attend to in everyday life including in cars where they are constrained in movement. Could intelligent cars sense their comfort levels for the purpose of maximizing it? To address this, first, we present a new dataset (available on request) of physical measures (skin temperature, blood volume pulse, electrodermal activity, and motion capture) and subjective thermal, sitting, and mental relaxation experience variables captured in semi-ecological settings in a car. Second, we provide an in-depth analysis of the relationship between passengers’ thermal experiences and physiological responses in the collected data. Our findings highlight complex duality in the relationship of thermal experience with heart rate variability and skin temperature variability. We discuss the practical implications that this may have for designing machine learning architectures for automatic detection of thermal discomfort.

Continuous assessment of task difficulty and mental workload is essential in improving the usability and accessibility of interactive systems. Eye tracking data has often been investigated to achieve this ability, with reports on the limited role of standard blink metrics. Here, we propose a new approach to the analysis of eye-blink responses for automated estimation of task difficulty. The core module is a time-frequency representation of eye-blink, which aims to capture the richness of information reflected on blinking. In our first study, we show that this method significantly improves the sensitivity to task difficulty. We then demonstrate how to form a framework where the represented patterns are analyzed with multi-dimensional Long Short-Term Memory recurrent neural networks for their non-linear mapping onto difficulty-related parameters. This framework outperformed other methods that used hand-engineered features. This approach works with any built-in camera, without requiring specialized devices. We conclude by discussing how Rethinking Eye-blink can benefit real-world applications.

Automatically monitoring and quantifying stress-induced thermal dynamic information in real-world settings is an extremely important but challenging problem. In this paper, we explore whether we can use mobile thermal imaging to measure the rich physiological cues of mental stress that can be deduced from a person's nose temperature. To answer this question we build i) a framework for monitoring nasal thermal variable patterns continuously and ii) a novel set of thermal variability metrics to capture a richness of the dynamic information. We evaluated our approach in a series of studies including laboratory-based psychosocial stress-induction tasks and real-world factory settings. We demonstrate our approach has the potential for assessing stress responses beyond controlled laboratory settings.

We introduce Deep Thermal Imaging, a new approach for close-range automatic recognition of materials to enhance the understanding of people and ubiquitous technologies of their proximal environment. Our approach uses a low-cost mobile thermal camera integrated into a smartphone to capture thermal textures. A deep neural network classifies these textures into material types. This approach works effectively without the need for ambient light sources or direct contact with materials. Furthermore, the use of a deep learning network removes the need to handcraft the set of features for different materials. We evaluated the performance of the system by training it to recognize 32 material types in both indoor and outdoor environments. Our approach produced recognition accuracies above 98% in 14,860 images of 15 indoor materials and above 89% in 26,584 images of 17 outdoor materials. We conclude by discussing its potentials for real-time use in HCI applications and future directions.

Mental stress is a critical problem in our modern society. This form of stress strongly affects our well being, and technology is needed to help us to manage health problems. The ability to automatically recognize a person's mental stress can be fundamental in supporting stress and health management. This research focuses on the use of mobile thermal imaging, a new and less explored sensor, to merge the measurement of multiple physiological signatures into one sensor and to build a reliable mental stress automatic recognition model. Mobile thermal imaging has greater potentials for real-world applications given that it is small and light weight, and requires low computation cost. To make mobile thermal imaging a robust multimodal stress sensor, we have so far contributed: i) a new robust respiration tracking method; and ii) a novel respiration-based automatic stress recognition model that works in ubiquitous settings. We are currently investigating new thermal signatures from underexplored body regions (i.e. trapezius muscle) and formulating a research framework to fuse multiple thermal signatures for more reliable stress recognition outcomes.

We propose DeepBreath, a deep learning model which automatically recognises people's psychological stress level (mental overload) from their breathing patterns. Using a low cost thermal camera, we track a person's breathing patterns as temperature changes around his/her nostril. The paper's technical contribution is threefold. First of all, instead of creating handcrafted features to capture aspects of the breathing patterns, we transform the uni-dimensional breathing signals into two dimensional respiration variability spectrogram (RVS) sequences. The spectrograms easily capture the complexity of the breathing dynamics. Second, a spatial pattern analysis based on a deep Convolutional Neural Network (CNN) is directly applied to the spectrogram sequences without the need of hand-crafting features. Finally, a data augmentation technique, inspired from solutions for over-fitting problems in deep learning, is applied to allow the CNN to learn with a small-scale dataset from short-term measurements (e.g., up to a few hours). The model is trained and tested with data collected from people exposed to two types of cognitive tasks (Stroop Colour Word Test, Mental Computation test) with sessions of different difficulty levels. Using normalised self-report as ground truth, the CNN reaches 84.59% accuracy in discriminating between two levels of stress and 56.52% in discriminating between three levels. In addition, the CNN outperformed powerful shallow learning methods based on a single layer neural network. Finally, the dataset of labelled thermal images will be open to the community.

We present RealPen, an augmented stylus for capacitive tablet screens that recreates the physical sensation of writing on paper with a pencil, ball-point pen or marker pen. The aim is to create a more engaging experience when writing on touch surfaces, such as screens of tablet computers. This is achieved by regenerating the friction-induced oscillation and sound of a real writing tool in contact with paper. To generate realistic tactile feedback, our algorithm analyzes the frequency spectrum of the friction oscillation generated when writing with traditional tools, extracts principal frequencies, and uses the actuator's frequency response profile for an adjustment weighting function. We enhance the realism by providing the sound feedback aligned with the writing pressure and speed. Furthermore, we investigated the effects of superposition and fluctuation of several frequencies on human tactile perception, evaluated the performance of RealPen, and characterized users' perception and preference of each feedback type.