The digital twin of humans is a relatively new concept. While many diverse definitions, architectures, and applications exist, a clear picture is missing on what, in fact, makes a human digital twin. Within this context, researchers and industrial use-case owners alike are unaware about the market potential of the-at the moment-rather theoretical construct. In this work, we draw a holistic vision of the human digital twin, and derive the specification of this holistic human digital twin in form of requirements, stakeholders, and users. For each group of users, we define exemplary applications that fall into the six levels of functionality: store, analyze, personalize, predict, control, and optimize. The functionality levels facilitate an abstraction of abilities of the human digital twin. From the manifold applications, we discuss three in detail to showcase the feasibility of the abstraction levels and the analysis of stakeholders and users. Based on the deep discussion, we derive a comprehensive list of requirements on the holistic human digital twin. These considerations shall be used as a guideline for research and industries for the implementation of human digital twins, particularly in context of reusability in multiple target applications.

Human and automation capabilities are the foundation of every human-autonomy interaction and interaction pattern. Therefore, machines need to understand the capacity and performance of human doing, and adapt their own behavior, accordingly. In this work, we address the concept of conjugated capabilities, i.e. capabilities that are dependent or interrelated and between which effort can be distributed. These may be used to overcome human limitations, by shifting effort from a deficient to a conjugated capability with performative resources. For example: A limited arm’s reach may be compensated by tilting the torso forward. We analyze the interrelation between elementary capabilities within the IMBA standard to uncover potential conjugation, and show evidence in data of post-rehabilitation patients. From the conjugated capabilities, within the example application of stationary manufacturing, we create a network of interrelations. With this graph, a manifold of potential uses is enabled. We showcase the graph’s usage in optimizing IMBA test design to accelerate data recordings, and discuss implications of conjugated capabilities on task allocation between the human and an autonomy.

Rescue robotics sets high requirements to perception algorithms due to the unstructured and potentially vision-denied environments. Pivoting Frequency-Modulated Continuous Wave radars are an emerging sensing modality for SLAM in this kind of environment. However, the complex noise characteristics of radar SLAM makes, particularly indoor, applications computationally demanding and slow. In this work, we introduce a novel radar SLAM framework, RaNDT SLAM, that operates fast and generates accurate robot trajectories. The method is based on the Normal Distributions Transform augmented by radar intensity measures. Motion estimation is based on fusion of motion model, IMU data, and registration of the intensity-augmented Normal Distributions Transform. We evaluate RaNDT SLAM in a new benchmark dataset and the Oxford Radar RobotCar dataset. The new dataset contains indoor and outdoor environments besides multiple sensing modalities (LiDAR, radar, and IMU).

Machines, e.g. empowered by AI and based on virtual models, can help to improve the quality of life. To exploit this potential and also integrate this with the real world and society, cooperation and teaming of these machines with humans, and with societies is crucial. Human-Machine Patterns can be a key concept to analyze, understand, design, engineer and evaluate the delicate interplay of humans and machines. Key issues here are to understand in which situations which agents should do, want to do, can do, will do and finally actually do which actions, and who then is accountable. This overview article is intended as an introduction into the special session on Shared and Cooperative Control, especially on patterns and models for controllability and resilience. It is a direct follow-up on the 2022's special session and overview article (which is also available on IEEE Xplore). It introduces the topic of shared and cooperative control of human-machine and human-AI systems, especially in the light of the new advances in AI technology. This paper gives a short overview on the state of research on interaction patterns, controllability and resilience, before it focuses on the fundamental aspects of which actor should do, wants to do, can do, does and will be accountable for a pattern, sub-pattern or action within a pattern. Examples of what can be achieved with this basic architecture are given, e.g. for the recognition of intent or for the support by assistant systems, using the automotive domain as a first application example.

Modern modelling approaches fail when it comes to understanding rather than pure supervision of human behavior. As humans become more and more integrated into human-to-anything interactions, the understanding of the human as a whole becomes critical. In this paper, we conduct a structured review of the human digital twin to indicate where modern paradigms fail to model the human agent. Particularly, the mechanistic viewpoint limits the usability of human and general digital twins. Instead, we propose a novel way of thinking about models, states, and their relations: Perspectives-Observer-Transparency. The modelling paradigm indicates how transparency - or whiteness - relates to the abilities of an observer, which again allows to model the penetration depth of a system model into the human psyche. The split in between the human's outer and inner states is described with a perspectives model, featuring the introperspective and the exteroperspective. We explore this novel paradigm by employing two recent scenarios from ongoing research and give examples to emphasize specific characteristics of the modelling paradigm.

The implicit assumption that human and autonomous agents have certain capabilities is omnipresent in modern teaming concepts. However, none formalize these capabilities in a flexible and quantifiable way. In this paper, we propose Capability Deltas, which establish a quantifiable source to craft autonomous assistance systems in which one agent takes the leader and the other the supporter role. We deduct the quantification of human capabilities based on an established assessment and documentation procedure from occupational inclusion of people with disabilities. This allows us to quantify the delta, or gap, between a team's current capability and a requirement established by a work process. The concept is then extended to the multidimensional capability space, which then allows to formalize compensation behavior and assess required actions by the autonomous agent.

To reach intuitive control of partially and highly automated machines, a smooth cooperation between both, human and machine, is necessary. A natural way to design cooperation is to use the structures of mental models already established in our minds by human-human or human-animal cooperation. These cooperation designs follow certain patterns to provide solutions in a variety of scenarios. By using interaction patterns, we form and trigger mental models in the human mind, which already uses patterns in a similar way. This paper provides a description of the structure of cooperation patterns. An exemplary pattern for the use case of control transition between a human and a driving automation is modelled based on data of an experiment of human drivers reacting to takeover requests. To use the patterns in human-machine cooperation systems, the novel concept of a pattern handler is introduced. The pattern handler can be instantiated in software and hardware, and matches the human behavior, automation actions, and environment data to cooperation patterns. Within the design phase, it helps with identifying and correcting flaws within the design.

As labor shortage is rising at an alarming rate, it is imperative to enable all people to work, particularly people with disabilities and elderly people. Robots are often used as universal tool to assist people with disabilities. However, for such human-robot workstations universal design fails. We mitigate the challenges of selecting an individualized set of input and output devices by matching devices required by the work process and individual disabilities adhering to the Convention on the Rights of Persons with Disabilities passed by the United Nations. The objective is to facilitate economically viable work-stations with just the required devices, hence, lowering overall cost of corporate inclusion and during redesign of workplaces. Our work focuses on developing an efficient approach to filter input and output devices based on a person's disabilities, resulting in a tailored list of usable devices. The methodology enables an automated assessment of devices compatible with specific disabilities defined in International Classification of Functioning, Disability and Health. In a mock-up, we showcase the synthesis of input and output devices from disabilities, thereby providing a practical tool for selecting devices for individuals with disabilities.

Cyber-Physical Systems constitute one of the core concepts in Industry 4.0 aiming at realizing production systems that combine the efforts of human workers, robots, and intelligent entities. This is particularly crucial in Human-Robot Collaboration manufacturing where a tight peer-to-peer interaction between humans and intelligent autonomous robots is necessary. The work proposes the integration of novel Artificial Intelligence technologies to enhance the flexibility and adaptability of collaborative robots. The integrated functionalities allow a collaborative robot to autonomously recognize the tasks a human worker performs, and accordingly adapt its behavior. The approach is deployed on a real HRC scenario showing the functioning of the developed cognitive capabilities and the increased flexibility of resulting collaborations.

Safety is a central challenge in human-robot collaboration. Particularly in higher collaboration levels, separating safety devices, such as fences, are no longer needed and must be replaced by intelligent sensor-based systems. Of particular interest is the adaptive speed control of the robot. This work presents a methodology to adaptively control the end-effector velocity of the robot based on the distances to dynamic environmental objects. The method combines distance measurement and environmental subtraction with conservative velocity estimation using robot-specific stopping distances and is available in real-time. Data acquisition is performed using a co-moving 3D camera sensor attached to the robot structure.

Human-robot collaboration offers the advantage of combining human characteristics and robotic capabilities, balancing individual weaknesses. In the inclusive project Next Generation, we are exploring the possibility of using collaborative robots as assistive devices for people with severe and multiple disabilities. An important step in implementing an inclusive workstation with a collaborative robot is determining the level of assistance required. Therefore, we developed a novel methodology and a capability-based software tool to determine the individual level of challenge. In this paper, we present the developed tool and its validation. We validate the methodology and tool using an industrial sample application from first labor market incorporating participants with varying mental and physical disabilities.