The global population is growing at an unprecedented pace, resulting in an increase in the number of people affected by cognitive decline. This paper presents a MR application for older adults with cognitive decline, aiming to stimulate cognitive and motor functioning and promote active ageing. Developed using Unity and deployed to HoloLens 2, the system allows users to interact with holograms through a series of engaging games. Before having elderly people with cognitive decline test the MR approach, preliminary studies were conducted to assess both its feasibility and usability. The results are encouraging, with participants reporting the approach is easy to learn and perform. They also felt confident and successful in accomplishing what they were asked to do. The immersive nature of MR has the potential to transform ageing into an experience filled with opportunities rather than limitations.

The integration of robots into human-shared environments has driven advancements in social robotics and natural Human-Robot Interaction. A key challenge in this field is to enable robots to interpret and respond to social cues, ensuring fluid, context-aware, and socially acceptable interactions. To address this, we propose a two-layer architecture for social navigation. The high-level reasoning layer utilizes Large Language Models to interpret contextual and environmental cues, generating constraints for navigation. These constraints are then enforced by the low-level layer, which employs Control Barrier Functions to ensure smooth and socially compliant robot movements. We validate our approach in a dynamic simulation environment, demonstrating effective constraint enforcement and socially acceptable navigation behavior.

In recent decades, the detection of fatigue has been largely studied to improve safety and performance in various domains such as healthcare, transportation, and manufacturing. In fact, fatigue significantly affects cognitive and physical performance. There are numerous studies on the detection of fatigue, but most of them focus on binary classification (i.e., fatigue vs. resting state) or different levels of fatigue intensity, without distinguishing between specific fatigue types.This work aims to discriminate between cognitive and physical fatigue, and proposes the use of physiological signals to classify four distinct conditions: rest, cognitive fatigue, physical fatigue, and combined cognitive and physical fatigue. We analyze data from cardiac, eye, electrodermal, and electromyographic activity. We consider different feature selection methods, including correlation analysis, principal component analysis and sequential forward floating search method. Ultimately, we classify them using state-of-the-art machine learning methods.The highest classification accuracy (86.823%) is obtained from a support vector machine method using a selection of the features extracted from cardiac and electromyographic sensors.This study highlights the potential for real-time fatigue classification, which can enhance the adaptability of automated systems to human needs, particularly in high risk environments like transportation and healthcare. Furthermore, the findings suggest that fatigue monitoring can be effectively conducted with minimal sensor requirements, contributing to the design of more efficient wearable sensor systems.

Collaborative robots (cobots) have seen widespread adoption in industrial applications over the last decade. Cobots can be placed outside protective cages and are generally regarded as much more intuitive and easy to program compared to larger classical industrial robots. However, despite the cobots’ widespread adoption, their collaborative potential and opportunity to aid flexible production processes seem hindered by a lack of training and understanding from shop floor workers. Researchers have focused on technical solutions, which allow novice robot users to more easily train collaborative robots. However, most of this work has yet to leave research labs. Therefore, training methods are needed with the goal of transferring skills and knowledge to shop floor workers about how to program collaborative robots. We identify general basic knowledge and skills that a novice must master to program a collaborative robot. We present how to structure and facilitate cobot training based on cognitive apprenticeship and test the training framework on a total of 20 participants using a UR10e and UR3e robot. We considered two conditions: adaptive and self-regulated training. We found that the facilitation was effective in transferring knowledge and skills to novices, however, found no conclusive difference between the adaptive or self-regulated approach. The results demonstrate that, thanks to the proposed training method, both groups are able to significantly reduce task time, achieving a reduction of 40%, while maintaining the same level of performance in terms of position error.Note to Practitioners—This paper was motivated by the fact that the adoption of smaller, so-called collaborative robots is increasing within manufacturing but the potential for a single robot to be used flexibly in multiple places of a production seems unfulfilled. If more unskilled workers understood the collaborative robots and received structured training, they would be capable of programming the robots independently. This could change the current landscape of stationary collaborative robots towards more flexible robot use and thereby increase companies’ internal overall equipment efficiency and competencies. To this end, we identify general skills and knowledge for programming a collaborative robot, which helps increase the transparency of what novices need to know. We show how such knowledge and skills may be facilitated in a structured training framework, which effectively transfers necessary programming knowledge and skills to novices. This framework may be applied to a wider scope of knowledge and skills as the learner progresses. The skills and knowledge that we identify are general across robot platforms, however, collaborative robot interfaces differ. Therefore, a practical limitation to the approach includes the need for a knowledgeable person on the specific collaborative robot in question in order to create training material in areas specific to that model. However, with our list of identified skills, it provides an easier starting point. We show that relatively few skills and knowledge areas can enhance a novice’s programming capability.

Industry 5.0 embodies the vision for the future of factories, emphasizing the importance of sustainable industrialization and the role of industry in society, through the key concept of placing the well-being of workers at the center of the production process. Building upon this vision, we propose a new paradigm to design human-centric industrial applications. To this end, we exploit Digital Twin (DT) technology to build a digital replica for each entity on the shop floor and support and augment interaction among workers and machines. While so far DTs in automation have been proposed for machine digitalization, the core element of the proposed approach is the Operator Digital Twin (ODT). In this scenario, biometrics allows to build a reliable model of those operator’s characteristics that are relevant in working contexts. Biometric traits are measured and processed to detect physical, emotional, and mental conditions, which are used to define the operator’s state. Perspectively, this allows to manage and monitor production and processes in an operator-in-the-loop manner, where not only is the operator aware of the state of the plant, but also any technological agent in the plant acts and reacts according to the operator’s needs and conditions. In this paper, we define the modeling of the envisioned ecosystem, present the designed DT’s blue-print architecture, discuss its implementation in relevant application scenarios, and report an example of implementation in a collaborative robotics scenario.Note to Practitioners—This paper was motivated by the problem of designing human-cyber-physical systems, where production processes are managed by concurrently taking into account operators, machines and plant status. This answers the needs of the novel Industry 5.0 paradigm, which aims to enhance social sustainability of modern factories. To this end, we propose an architecture based on digital twins that allows to develop a digital layer, detached from the physical one, where the plant can be monitored and managed. This allows the creation of a digital ecosystem where machines, operators, and the interactions among them are represented, augmented, and managed. We discuss how the proposed architecture can be applied to three relevant scenarios: remote training and maintenance, line operation and line supervision. Moreover, the implementation in a collaborative robotics scenario is presented, to provide an example of the proposed architecture can be implemented in industrial scenarios.

This paper proposes a groundbreaking architecture that reimagines Industry 5.0, emphasizing human-centric technological integration via the Web of Things (WoT) standard. Our approach innovatively digitizes human operators and machinery, creating a responsive industrial ecosystem attentive to real-time human conditions. Central to this is the Operator Thing (OT), a digital replica representing the human operator's status and needs. This system not only recognizes operator stress and discomfort but intelligently adjusts, ensuring optimal human-machine synergy. Our methodology extends to redefining operational parameters and tasks in response to human states, balancing well-being with production efficiency. The ultimate goal is a transformative, adaptive, and empathetic Industry 5.0 environment, validated through rigorous interdisciplinary evaluation.

In the era of Industry 4.0, the importance of human-robot collaboration (HRC) in the advancement of modern manufacturing and automation is paramount. Understanding the intricate physiological responses of the operator when they interact with a cobot is essential, especially during programming tasks. To this aim, wearable sensors have become vital for real-time monitoring of worker well-being, stress, and cognitive load. This article presents an innovative dataset (SenseCobot) of physiological signals recorded during several collaborative robotics programming tasks. This dataset includes various measures like ElectroCardioGram (ECG), Galvanic Skin Response (GSR), ElectroDermal Activity (EDA), body temperature, accelerometer, ElectroEncephaloGram (EEG), Blood Volume Pulse (BVP), emotions and subjective responses from NASA-TLX questionnaires for a total of 21 participants. By sharing dataset details, collection methods, and task designs, this article aims to drive research in HRC advancing understanding of the User eXperience (UX) and fostering efficient, intuitive robotic systems. This could promote safer and more productive HRC amid technological shifts and help decipher intricate physiological signals in different scenarios.

Fatigue is a multifaceted construct, that represents an important part of human experience. The two main aspects of fatigue are the mental one and the physical one, that often intertwine, intensifying their collective impact on daily life and overall well-being. To soften this impact, understanding and quantifying fatigue is crucial. Physiological data play a pivotal role in the comprehension of fatigue, allowing a precious insight into the level and type of fatigue experienced. Though the analysis of these biosignals, researchers can determine whether the person is feeling mental fatigue, physical fatigue or a combination of both. This paper introduces MePhy, a comprehensive dataset containing various biosignals, gathered while inducing different fatigue conditions, in particular mental and physical fatigue. Among the biosignals closely associated with stress situations, we chose: eye activity, cardiac activity, electrodermal activity (EDA) and electromyography (EMG). Data were collected using different devices, including a camera, a chest strap and different sensors from the BITalino kit.

Mass customization and shorter manufacturing cycles are becoming more important among small and medium-sized companies. However, classical industrial robots struggle to cope with product variation and dynamic environments. In this paper, we present CoBT, a collaborative programming by demonstration framework for generating reactive and modular behavior trees. CoBT relies on a single demonstration and a combination of data-driven machine learning methods with logic-based declarative learning to learn a task, thus eliminating the need for programming expertise or long development times. The proposed framework is experimentally validated on 7 manipulation tasks and we show that CoBT achieves ≈ 93% success rate overall with an average of 7.5s programming time. We conduct a pilot study with non-expert users to provide feedback regarding the usability of CoBT. More videos and generated behavior trees are available at: https://github.com/jainaayush2006/CoBT.git.

Recently, electroencephalographic (EEG) signals have been used to enhance Human-Robot Interaction (HRI). In particular, Error-Related Potentials (ErrPs) have been exploited since very few years. These potentials are evoked when there is a mismatch between the command given by the subject and the movements of the robot, or if the user’s expectation is different from the robot or other human behavior. These signals can be used to improve and customize the robot system, as feedback to better adapt the robot to human needs. This work aims to investigate and detect the ErrPs during different interaction tasks. We set up an experiment divided into five different tasks, where every task has 120 events with a 25%-35% probability of error. The robot used in the experiment is a Baxter robot and the commands from the subject to the robot are sent in two different ways: with a keyboard or with a motion capture device. This work aims to reproduce a simplified teleoperated pick and place task. However, the achieved results do not allow to correctly identify the ErrPs, but exhibit only some minor differences between trials with and without errors. Hence, we here analyze the reasons behind such negative results, focusing on the challenges of the structure and the setup of the experiment. We analyze the possible problems and provide some recommendations to overcome them in similar use cases.

Recently, electroencephalographic (EEG) signals have been used to design and enhance human-robot interaction (HRI). In particular, error-related potentials (ErrPs) have been leveraged since very few years. These potentials can be used to provide feedback to the robot about any mismatch between the user’s expectations and the robot’s behavior, during interaction tasks. In this process, the correct classification of ErrPs is crucial, which, in turn, relies on the reliability of the process for extraction and selection of signal features. In this work, we consider an extensive list of possible features and perform a statistical analysis to assess their discriminative power for ErrP analysis. The aim is to reduce the number of features used for classification while retaining the most relevant ones only. Overall, the outcome of our study shows that some parameters have relevant importance compared to others, (i.e. temporal features, frequency features, signal processing features, and some wavelet transform coefficients), while some of the features used in existing works are not useful since they have low discriminative power.

The growing spread of robots for service and industrial purposes calls for versatile, intuitive and portable interaction approaches. In particular, in industrial environments, operators should be able to interact with robots in a fast, effective, and possibly effortless manner. To this end, reality enhancement techniques have been used to achieve efficient management and simplify interactions, in particular in manufacturing and logistics processes. Building upon this, in this paper we propose a system based on mixed reality that allows a ubiquitous interface for heterogeneous robotic systems in dynamic scenarios, where users are involved in different tasks and need to interact with different robots. By means of mixed reality, users can interact with a robot through manipulation of its virtual replica, which is always colocated with the user and is extracted when interaction is needed. The system has been tested in a simulated intralogistics setting, where different robots are present and require sporadic intervention by human operators, who are involved in other tasks. In our setting we consider the presence of drones and AGVs with different levels of autonomy, calling for different user interventions. The proposed approach has been validated in virtual reality, considering quantitative and qualitative assessment of performance and user's feedback.

Industry 5.0 promotes the development of human-centered industrial operations fueled by a fresh wave of disruptive technologies that encourage synergistic human-machine integration. Its focus is on understanding how human cognition contributes to a more secure and harmonious coexistence between humans and machines in industrial scenarios, employing solutions that prioritize fundamental worker demands while preserving or enhancing industrial productivity. In this context, the ability to assess fatigue objectively is crucial for occupational health and safety because it can reduce cognitive and motor function, ultimately lowering productivity and raising the risk of harm to human operators. To this end, wearable systems provide a promising solution for continuous, non-intrusive, and long-term monitoring of biological signals for fatigue detection. However, the adoption of these devices presents unique challenges, such as inter-individual variability that renders traditional one-size-fits-all machine learning models unsuitable. This paper provides an analysis of the current state-of-the-art for wearable device monitoring, including ongoing issues and current knowledge gaps. In addition, an experimental analysis is presented, employing a pattern discovery pipeline based on unsupervised learning on a real-world dataset. Our analysis provides experimental evidence of the limitations of one of the classical approaches to fatigue assessment, thus highlighting the need for more advanced models.

Recent advances in robotics have allowed the introduction of robots assisting and working together with human subjects. To promote their use and diffusion, intuitive and user-friendly interaction means should be adopted. In particular, gestures have become an established way to interact with robots since they allow to command them in an intuitive manner. In this article, we focus on the problem of gesture recognition in human–robot interaction (HRI). While this problem has been largely studied in the literature, it poses specific constraints when applied to HRI. We propose a framework consisting in a pipeline devised to take into account these specific constraints. We implement the proposed pipeline considering, as an example, an evaluation use case. To this end, we consider standard machine learning algorithms for the classification stage and evaluate their performance considering different performance metrics for a thorough assessment.

Building on the benefits of Industry 4.0, Industry 5.0 promotes human-centricity of factories, placing operator’s wellbeing at the center of production. Production processes are expected to be designed around operator’s needs, for a more sustainable contribution of industry to society. In this broad context, in this paper we consider the problem of monitoring operator’s condition and, specifically, detecting any mental or physical fatigue they might be experiencing in workplaces. Indeed, if the onset of any source of fatigue is monitored, it is possible to introduce assistive strategies that preserve productivity, on the one side, and operator’s wellbeing, on the other side. To achieve this goal, we detect mental and physical fatigue conditions via physiological monitoring, by means of a wearable device that measures cardiac activity. We design an experimental protocol such that test participants are exposed to mental and physical fatigue. Their heart rate variability is then extracted and analysed to discriminate among rest, mental fatigue, physical fatigue and joint mental and physical fatigue. The achieved results show that statistically significant differences can be found in time-domain metrics. Moreover, the analysis of the empirical distribution functions shows, for each metrics, the conditions that exhibit the greatest differences and, hence, that can be distinguished more accurately. However, results show also that, in the presence of physical fatigue, it is difficult to detect the presence of additional mental fatigue.

The key concept of collaborative robotics is represented by the presence of a strict interaction between a human user and the robotic system. As such, the study of the interaction is of paramount importance for a successful implementation of the system. In this article, we propose a novel approach to address the problem of designing a collaborative robotic system for industrial applications, focusing on the characteristics of the interaction. In particular, we will propose a set of methodologies focused on interaction design, inspired by those used for the design of user interfaces. These methodologies will allow the design of collaborative robotic systems following a user-centered approach, thus putting emphasis not only on safety and adaptability of the robotic systems (which have been widely addressed in the literature), but also on the interaction experience. While the proposed methodology was developed considering general collaborative robotics applications, two real industrial case studies were considered, to instantiate the considered framework and showcase its applicability to the real-world domain. Note to Practitioners—This article aims at bridging the gap between interaction design and collaborative robotics. In particular, the proposed methodology will represent a toolset for robotic experts (researchers and system integrators), for understanding the user experience and designing the robotic system ensuring an effective interaction. In fact, while robotics experts are typically well aware of issues and methodologies related to technological and application aspects, they often tend to ignore the principles of interaction. Such principles are commonly adopted in the design of computer-based human–machine interfaces or web applications, but, to the best of the authors’ knowledge, have never been applied to the design of collaborative robotic systems for industrial applications. Hence, this article will serve as a fundamental step to bring interaction design principles into the robot integration domain.

In recent years, production systems have become highly sophisticated and complex. As a result, while on the one hand, the least-skilled labor has been partially displaced by machines, high-skilled labor is more required to supervise and control advanced automation systems. In many cases, the complexity of machines implies an increased complexity of human–machine interfaces (HMIs), which are the main point of contact between the operator and the machine. To enable effective use of HMIs and to enable their usage by workers with different knowledge and capabilities, novel design approaches have been proposed. In particular, in this article, we consider the approach developed in the framework of the European research project INCLUSIVE, which aimed at designing industrial HMIs that adapt to the skills and capabilities of human operators. As a case study, we consider an adaptive interaction system for the woodworking industry and present an extensive evaluation carried out in real production environment with shopfloor workers. The effectiveness of the INCLUSIVE approach has been assessed with subjective and objective measurements and compared to that of interaction systems customarily used in industry. Results have shown that users appreciated the INCLUSIVE system and largely preferred it over the customary system. Moreover, with regard to objective performance-related measurements, they performed better when using the INCLUSIVE system since they received tailored guidance during the considered working tasks. Note to Practitioners— This article was motivated by the fact that advanced automation systems are often highly complex for human operators. To address this problem, we focus on the importance of designing the automation system around users and discuss an approach for adaptive automation, called INCLUSIVE. Its main feature is that it adapts the human–machine interface (HMI) according to operator’s skills, capabilities, and current mental fatigue. In this article, we provide an extensive evaluation of the INCLUSIVE system, considering a company producing woodworking machines as a use case. Assessment was carried out in the company shopfloor, considering real workers, and the INCLUSIVE system was compared with the customary HMI running on the company machines. The results of our study suggest that adapting the interaction to operator’s needs allows better working performance while letting workers more satisfied with the use of the system.

While modern production systems are becoming increasingly technologically advanced, the presence of human operators remains fundamental in industrial workplaces. To complement and enhance the capabilities of human workers, approaches based on adaptive automation have been introduced. They consist of adapting the behavior of the system according to the user’s capabilities and effort. In this article, we present a general holistic framework for adaptive automation, called INCLUSIVE, that assists the operator during working tasks. The system consists of three modules. First, a thorough characterization of the operator’s constitutional and situational condition is provided; based on this, properly tailored adaptation is given, and if necessary, further training and support are provided. The framework has been implemented and tested considering three industrial use cases, selected as representative of a wide area of interest for the industry in Europe, in terms of both production requirements and involved operators. Tests have been carried out in real production environments, considering real production tasks carried out by 53 shop-floor workers. Results have shown that workers’ satisfaction when using the INCLUSIVE system and their performances was increased with respect to customary interaction systems currently used in industries. Moreover, the achieved results were used to formulate a set of recommendations for the design and implementation of an adaptive interaction system in relation to ensuring worker satisfaction and system usability in an industrial environment, as well as performance requirements.Note to Practitioners—This article was motivated by the fact that, despite modern advanced automation, human operators are still central in the manufacturing process. However, technological progress often causes challenging interaction with complex industrial systems. The goal of this article is to introduce a complete framework for adaptive automation, with the ultimate goal of facilitating the interaction of human operators with complex industrial systems. The framework relies on three modules: measurement of human capabilities, the adaption of the interaction system, and additional teaching and support. The three modules are discussed at a high level, independently of the target application. Moreover, to facilitate their application in specific working contexts, examples are provided with respect to three different industrial applications. Results of tests carried out with shop-floor operators show that implementing the proposed framework allows better working performance and increases worker satisfaction with the use of automation.

Modern production systems are becoming more and more complex to comply with diversified market needs, flexible production, and competitiveness. Despite technological progress, the presence of human operators is still fundamental in production plants, since they have the important role of supervising and monitoring processes, by interacting with such complex machines. The complexity of machines implies an increased complexity of human-machine interfaces (HMIs), which are the main point of contact between the operator and the machine. Thus, HMIs cannot be considered anymore an accessory to the machine and their improvement has become an important part of the design of the whole machines, to enable a nonstressful interaction and make them easy to also use less skilled operators. In this article, we present a general framework for the design of HMIs that adapt to the skills and capabilities of the operator, with the ultimate aim of enabling a smooth and efficient interaction and improving user's situation awareness. Adaptation is achieved by considering three different levels: perception (i.e., how information is presented), cognition (i.e., what information is presented), and interaction (i.e., how interaction is enabled). For each level, general guidelines for adaptation are provided, thus defining a meta-HMI independent of the application. Finally, some examples of how the proposed adaptation patterns can be applied to the case of procedural and extraordinary maintenance tasks are presented.

Legibility can be defined as the ability of a robot to communicate its intent to the user. Legibility is relatively little investigated in multi-robot systems, but, in the literature, studies exist where the trajectory of manipulators is analyzed as a factor to improve the collaboration between robots and users. In this paper, we focus on the legibility of a group of mobile robots. To this end, we consider a set of motion-variables: trajectory, dispersion and stiffness. They are typical parameters that determine the motion of a group of robots. To analyze the effect of the motion-variables over legibility, Fisher's exact test and ANOVA (analysis of variance) were carried out. The data for the statistical analysis cover a full factorial plan and they were collected in a virtual reality set-up, where the users shared the environment with a group of robots. We investigate two aspects of legibility: the correctness and the rapidity of communication, namely if the communication happens correctly and how fast it happens. Trajectory was found to be relevant to correctly communicate the intention of the robots, while stiffness and dispersion were relevant for the rapidity of legibility.

Human-robot interaction has gained a lot of attention in recent years, since the use of robots can complement and improve human capabilities. To make such interaction smooth, proper interaction approaches are needed. Customarily these are tested in simplified scenarios and tame laboratory environment, since reproducing complex real use cases is often difficult. Achieved results are then not representative of actual interaction in reality and do not scale to complex scenarios. To overcome this issue, in this paper we consider the use of virtual reality as an alternative tool to assess HRI in those scenarios that are difficult to reproduce in reality. To this end, we compare the interaction experience for the same task, which is carried out in both virtual reality and real environment. To assess user's interaction in the two scenarios, we consider quantitative task related metrics, mental workload sustained, and subjective reporting. Results show that virtual reality allows to reproduce a faithful interaction experience and, thus, can be used to reliably validate human-robot interaction approaches in complex scenarios.

In this paper we present a general framework for affective human-robot interaction that allows users to intuitively interact with a robot and takes into account their mental fatigue, thus simplifying the task or providing assistance when the user feels stressed. Interaction with the robot is achieved by naturally mapping user's forearm motion, detected with a smartwatch, into robot's motion. High-level commands can be provided by means of gestures. An approach based on affective robotics is used to adapt the level of robot's autonomy to the cognitive workload of the user. User's mental fatigue is detected from the analysis of heart rate, also measured by the smartwatch. The framework is general and can be applied to different robotic systems. In this paper, we consider its experimental validation on a wheeled mobile robot.

Modern manufacturing systems typically require high degrees of flexibility, in terms of ability to customize the production lines to the constantly changing market requests. For this purpose, manufacturing systems are required to be able to cope with changes in the types of products, and in the size of the production batches. As a consequence, the human-machine interfaces (HMIs) are typically very complex, and include a wide range of possible operational modes and commands. This generally implies an unsustainable cognitive workload for the human operators, in addition to a non-negligible training effort. To overcome this issue, in this paper we present a methodology for the design of adaptive human-centred HMIs for industrial machines and robots. The proposed approach relies on three pillars: measurement of user's capabilities, adaptation of the information presented in the HMI, and training of the user. The results expected from the application of the proposed methodology are investigated in terms of increased customization and productivity of manufacturing processes, and wider acceptance of automation technologies. The proposed approach has been devised in the framework of the European project INCLUSIVE.