Machines, e.g. empowered by AI and based on virtual models, can help to improve the quality of life. To exploit this potential and also integrate this with the real world and society, cooperation and teaming of these machines with humans, and with societies is crucial. Human-Machine Patterns can be a key concept to analyze, understand, design, engineer and evaluate the delicate interplay of humans and machines. Key issues here are to understand in which situations which agents should do, want to do, can do, will do and finally actually do which actions, and who then is accountable. This overview article is intended as an introduction into the special session on Shared and Cooperative Control, especially on patterns and models for controllability and resilience. It is a direct follow-up on the 2022's special session and overview article (which is also available on IEEE Xplore). It introduces the topic of shared and cooperative control of human-machine and human-AI systems, especially in the light of the new advances in AI technology. This paper gives a short overview on the state of research on interaction patterns, controllability and resilience, before it focuses on the fundamental aspects of which actor should do, wants to do, can do, does and will be accountable for a pattern, sub-pattern or action within a pattern. Examples of what can be achieved with this basic architecture are given, e.g. for the recognition of intent or for the support by assistant systems, using the automotive domain as a first application example.

The implicit assumption that human and autonomous agents have certain capabilities is omnipresent in modern teaming concepts. However, none formalize these capabilities in a flexible and quantifiable way. In this paper, we propose Capability Deltas, which establish a quantifiable source to craft autonomous assistance systems in which one agent takes the leader and the other the supporter role. We deduct the quantification of human capabilities based on an established assessment and documentation procedure from occupational inclusion of people with disabilities. This allows us to quantify the delta, or gap, between a team's current capability and a requirement established by a work process. The concept is then extended to the multidimensional capability space, which then allows to formalize compensation behavior and assess required actions by the autonomous agent.

To reach intuitive control of partially and highly automated machines, a smooth cooperation between both, human and machine, is necessary. A natural way to design cooperation is to use the structures of mental models already established in our minds by human-human or human-animal cooperation. These cooperation designs follow certain patterns to provide solutions in a variety of scenarios. By using interaction patterns, we form and trigger mental models in the human mind, which already uses patterns in a similar way. This paper provides a description of the structure of cooperation patterns. An exemplary pattern for the use case of control transition between a human and a driving automation is modelled based on data of an experiment of human drivers reacting to takeover requests. To use the patterns in human-machine cooperation systems, the novel concept of a pattern handler is introduced. The pattern handler can be instantiated in software and hardware, and matches the human behavior, automation actions, and environment data to cooperation patterns. Within the design phase, it helps with identifying and correcting flaws within the design.

Future automated driving systems (ADS) are discussed as having the ability to “override” driver control inputs. Yet, little is known about how drivers respond to this, nor how a human-machine interaction (HMI) for them should be designed. This work identifies intervention types associated with an ADS that has change control authority and outlines an experiment method which simulates a deficit in driver situation awareness, enabling the study of their responses to interventions in a controlled environment. In a simulator study (N = 18), it was found that drivers express more negative valence when their control input is blocked (p = .046) than when it is taken away. In safety-critical scenarios, drivers respond more positively to interventions (p = .021) and are willing to give the automation more control (p = .018). An experimental method and HMI design insights are presented and ethical questions about the development of automated driving are provoked.

With increasing autonomous capabilities and intelligence of machines and interconnectivity of humans and machines, interaction and cooperation become even more crucial for socio-cyber-physical systems. How can we organize the exploding number of design options in a way that it is easy for humans to understand and control these systems? Patterns for the design of interaction and cooperation could be the key for describing problems and generalizable solution patterns, which can be instantiated again in individual solutions. Modeling, matching and instantiating could be the fundamental use cases for those patterns. Understanding, designing, engineering and using are the essential activities in an interplay of researchers, designers, engineers and users of human-machine systems. A realistic dream is to have global databases of human-machine patterns and their effects so that the wheel does not have to be re-invented several times and common knowledge about patterns can be shared in our community.