Detecting and segmenting dysfluencies is crucial for effective speech therapy and real-time feedback. However, most methods only classify dysfluencies at the utterance level. We introduce StutterCut, a semi-supervised framework that formulates dysfluency segmentation as a graph partitioning problem, where speech embeddings from overlapping windows are represented as graph nodes. We refine the connections between nodes using a pseudo-oracle classifier trained on weak (utterance-level) labels, with its influence controlled by an uncertainty measure from Monte Carlo dropout. Additionally, we extend the weakly labelled FluencyBank dataset by incorporating frame-level dysfluency boundaries for four dysfluency types. This provides a more realistic benchmark compared to synthetic datasets. Experiments on real and synthetic datasets show that StutterCut outperforms existing methods, achieving higher F1 scores and more precise stuttering onset detection.

As robots increasingly permeate diverse domains like healthcare, education, service industries and homes, accurately understanding humans’ responses to and behaviour towards robots is crucial. While many human-robot interaction (HRI) studies focus on either quantitative or qualitative approaches, we advocate a mixed-method approach. This study investigated robot norm violations by implementing a scenario where a mobile manipulator robot and a human, in-person, carry out a physical, competitive task. Sixty-two participants were recruited and randomly assigned to either an experimental or a control condition (balanced for age/gender). The scenario was a competitive scavenger hunt game where participants took turns with a robot. We investigated the robot behaviours’ effects on trust, discomfort, competence, enjoyment, participant behaviour and physiological changes. The mixed-method approach integrated physiological measurements, behavioural observations and qualitative responses, thus offering a comprehensive account of HRI dynamics in the context of norm violations. Questionnaire results reveal significant shifts in human perceptions and attitudes when social norms are violated by robots, compared to a norm-compliant control condition. Specifically, trust and enjoyment decrease, discomfort increases and the robot’s perceived competence is compromised. These findings are extended through additional analyses of participants’ physiological changes, behaviours and responses to open-ended questions. Behavioural observations indicated increased verbal engagement and emotional responses, while physiological data showed elevated stress levels in the experimental group. Our study highlights the advantage of a mixed-methods approach combining different qualitative and quantitative data, providing a more comprehensive picture of participants’ perceptions of a robot, and how they react and respond to robot norm violations.

As robots integrate more into daily life, socially aware robots with specific social attributes and behaviors are necessary. This review aims to explore how social norms in Human–Robot Interaction (HRI) impact robot design and human perception. We searched for relevant articles in the following databases: ACM Digital Library, IEEE Digital Library, Scopus, Springer Link, and PsycINFO. After applying inclusion and exclusion criteria, a final set of 69 articles were included in the review. These articles were categorized based on whether they examined norm conformity or norm violations, and were further sorted into 12 categorical norm labels to assist in analysis and comparison. By examining the existing literature, this review uncovers how social norms impact aspects of HRIs like trust, acceptance, and comfort while highlighting the importance of aligning robot design with user expectations. It reveals design challenges such as accounting for cultural variations, context-specific norms, and evolving norms over time. Addressing these challenges has the potential to improve user experiences, promote broader acceptance of robots, and foster successful integration of robots into various domains. The findings contribute to the ongoing discussion on the role of social norms in HRI, offering valuable insights and a foundation for future research.

Prediction of transition between locomotion modes (e.g. moving from flat ground to stairs, etc) is vital for optimal interface with lower limb assistive technologies such as exoskeletons and prostheses. Inertial and bipolar electromyography (EMG) sensors have been investigated, but accuracy for clinical utility remains unresolved. This shortfall may be attributed to their limited capacity to detect subtle changes in muscle activations, particularly during the early stages of locomotion transitions (e.g., near the toe-off). In this study, we examined the effectiveness of two high-density surface electromyography (HDsEMG) sensors in detecting muscle activation changes during stair-related transitions. The results revealed that compared to bipolar EMG on the same muscles, HDsEMG-based methods increased transition prediction accuracy significantly from 70.2% to 91.1% when predicting at toe-off and from 89.8% to 99.2% when predicting with a delay of 400-ms relative to toe-off. This demonstrated the superior ability of HDsEMG to capture subtle muscle activation changes, especially during early transition stages. We also found reducing the electrode count to 21 per muscle only minimally impacted performance (88.3% accuracy at toe-off). This suggests distributing the same total number of electrodes across more muscles could potentially further improve prediction accuracy without increasing computational load. Moreover, by implementing image-inpainting signal processing, HDsEMG demonstrated robustness against the common issue of electrode signal loss. Even with 30% electrode detachment, prediction accuracy decreased only by 3%. We argue that HDsEMG offers a promising solution to bridge the gap in locomotion transition prediction for interface with assistive technology.

Speech anonymisation aims to protect speaker identity by changing personal identifiers in speech while retaining linguistic content. Current methods fail to retain prosody and unique speech patterns found in elderly and pathological speech domains, which is essential for remote health monitoring. To address this gap, we propose a voice conversion-based method (DDSP-QbE) using differentiable digital signal processing and query-by-example. The proposed method, trained with novel losses, aids in disentangling linguistic, prosodic, and domain representations, enabling the model to adapt to uncommon speech patterns. Objective and subjective evaluations show that DDSP-QbE significantly outperforms the voice conversion state-of-the-art concerning intelligibility, prosody, and domain preservation across diverse datasets, pathologies, and speakers while maintaining quality and speaker anonymity. Experts validate domain preservation by analysing twelve clinically pertinent domain attributes.

The successful integration of domestic service robots into home environments can bring significant services and convenience to the general population and possibly mitigate important societal issues, such as care provision for older adults. However, home environments are complex, dynamic and object-rich. It is, thus, very probable that service robots will encounter ambiguity while interacting with household items. To enable service robots to be more adaptive, we proposed a learning so-cial referencing computational framework and experimentally evaluated the framework on a mobile manipulator robot, Fetch, in object selection scenarios. The framework allows the robot to (1) detect and analyze the ambiguity level based on the robot's view and user's command, (2) assess the human's attention level and attract their attention, (3) disambiguate references to objects using human feedback and (4) learn novel objects after clarification from the user. System evaluation results are presented. The framework is modular and can be applied to different robotic platforms.

Between 2% to 5% of children are affected by Developmental Coordination Disorders in Canada and have been diagnosed with upper limb impairments, which affect their daily lives and reduces their autonomy. Motor impairments can be part of progressive disorders, so despite regular therapy, progress remains fleeting. Affected individuals therefore consistently face many barriers, including entertainment opportunities, as availability of off-the-shelf inclusive technology is very limited. Our long-term goal is to develop a play-mediator robot, which would facilitate play between children with motor impairments and their peers or family members. Here, games that the robot can play are remotely controlled by the participants, using appropriate interfaces (e.g. joysticks). In this paper, we take the first step towards that goal and develop an adaptive joystick controller that can compensate for individual deficits. We monitor movement statistics to determine if re-calibration of the controller is necessary. Moreover, we propose a computational model of data ‘distortion’, as a tool for developers to test their technology in the very early stages of prototype development, without requiring access to participants. This work is validated with data from healthy adults and children with upper limb impairments.

Speech/voice disorders are common in People Living with Dementia (PLwD). Fluctuations in speech quality can serve as biomarkers of cognitive deterioration but there is a gap in automated assessment of speech collected in unstructured environs. Our organisation has deployed Alexa in the households of 14 PLwD to track self-reported mental and physical state as well as use of language. n this work, we present a case study analysing highly variable speech over time, providing potential insights into cognitive changes. Alexa data gathered from the participant was manually annotated with speech assessment labels. Those labels are matched to openSMILE features by performing a feature importance analysis to isolate critical features that contribute to the perceptual ratings. We can assess phonation with a F1-score of 0.55, breathiness: 0.71, roughness: 0.60, asthenia: 0.65, strain: 0.74. This work is a first step towards automatic speech assessment to monitor cognitive impairment over time.

Play is valuable in making therapy more enjoyable, and has been studied intensively in human-robot interaction. However, the use of robots as play-mediators in multiplayer games, and the study of the dynamics of players have barely been explored. In this work, pairs of participants played with the MyJay robot in a game with two collaborative conditions (Shared and Fusion). In the Shared condition, participants shared the tasks and in the Fusion condition, participants had to synchronize their commands for the robot. In previous work, we analyzed the video recordings and questionnaires and observed that participants perceived the Fusion condition as more challenging, and requiring more coordination, while the Shared condition was perceived as more enjoyable. This paper will report on new analyses based on physiological and joystick data. The results revealed different patterns of heart rate and usage of the joysticks in the two conditions, while no link between physiological data and enjoyment was found.

This study aimed to investigate the effects of norm-violating behaviour on human perception and attitudes towards robots. Specifically, we examined the impact of a robot performing social norm violations in the context of a competitive scavenger hunt game. During the game, the robot was programmed to engage in predefined behaviours considered as social norm violations, including both injunctive and descriptive norm violations (e.g., cheating, and making loud noises). The study used an experimental and control group, with participants either exposed to norm-violating behaviour or not, respectively. The results indicated that participants in the experimental group had a strong awareness of the norm-violating behaviour according to self-reported assessments. Additionally, post-questionnaire results revealed a significant difference in trust, overall enjoyment, and discomfort between the two groups. These findings show that in our study, participants expected robots to abide by both types of social norms (i.e., injunctive and descriptive) and that violations of them negatively impacted participants’ perceptions and attitudes towards robots. This further emphasizes the importance of considering social norms in the design and programming of robots for human-robot interactions.

Advancements in conversational AI have created unparalleled opportunities to promote the independence and well-being of older adults, including people living with dementia (PLWD). However, conversational agents have yet to demonstrate a direct impact in supporting target populations at home, particularly with long-term user benefits and clinical utility. We introduce an infrastructure fusing in-home activity data captured by Internet of Things (IoT) technologies with voice interactions using conversational technology (Amazon Alexa). We collect 3103 person-days of voice and environmental data across 14 households with PLWD to identify behavioural patterns. Interactions include an automated well-being questionnaire and 10 topics of interest, identified using topic modelling. Although a significant decrease in conversational technology usage was observed after the novelty phase across the cohort, steady state data acquisition for modelling was sustained. We analyse household activity sequences preceding or following Alexa interactions through pairwise similarity and clustering methods. Our analysis demonstrates the capability to identify individual behavioural patterns, changes in those patterns and the corresponding time periods. We further report that households with PLWD continued using Alexa following clinical events (e.g., hospitalisations), which offers a compelling opportunity for proactive health and well-being data gathering related to medical changes. Results demonstrate the promise of conversational AI in digital health monitoring for ageing and dementia support and offer a basis for tracking health and deterioration as indicated by household activity, which can inform healthcare professionals and relevant stakeholders for timely interventions. Future work will use the bespoke behavioural patterns extracted to create more personalised AI conversations.

Stuttering detection and classification are important issues in speech therapy as they could help therapists track the progression of patients’ dysfluencies. This is also an important tool for technology-assisted speech therapy. In this paper, we combine MFCC and phoneme probabilities to train a neural network for stuttering detection and classification of four dysfluency types. We evaluate our system on the UCLASS, FluencyBank and SEP-28K datasets and show that our system is effective and suitable for real-time applications.

Complex tasks require the combination of both discrete and rhythmic movements. Though scientists do not yet agree on the neural architecture involved in both types and in the transition from one to the other, the importance of having robot controllers able to behave rhythmically and discretely is universally recoanized. In this paper, a bio-inspired robot controller based on oscillating neurons is proposed to realize both discrete and rhythmic movements and easily transition from one to the other. It is shown that, under certain parameter conditions, the CPG controller behaves like a PID controller. In order to demonstrate the feasibility of controlling both discrete and rhythmic movements, the CPG is applied to the initiation of handshaking, namely, reach towards the human hand and start to shake it. Results show that this architecture is suitable for both discrete and rhythmic movements and can easily transition from one to the other.