Machines, e.g. empowered by AI and based on virtual models, can help to improve the quality of life. To exploit this potential and also integrate this with the real world and society, cooperation and teaming of these machines with humans, and with societies is crucial. Human-Machine Patterns can be a key concept to analyze, understand, design, engineer and evaluate the delicate interplay of humans and machines. Key issues here are to understand in which situations which agents should do, want to do, can do, will do and finally actually do which actions, and who then is accountable. This overview article is intended as an introduction into the special session on Shared and Cooperative Control, especially on patterns and models for controllability and resilience. It is a direct follow-up on the 2022's special session and overview article (which is also available on IEEE Xplore). It introduces the topic of shared and cooperative control of human-machine and human-AI systems, especially in the light of the new advances in AI technology. This paper gives a short overview on the state of research on interaction patterns, controllability and resilience, before it focuses on the fundamental aspects of which actor should do, wants to do, can do, does and will be accountable for a pattern, sub-pattern or action within a pattern. Examples of what can be achieved with this basic architecture are given, e.g. for the recognition of intent or for the support by assistant systems, using the automotive domain as a first application example.

The current advancement of technology with not only physical, but increasing cognitive functions widely coined as Artificial Intelligence (AI) leads to multiple situations where humans willingly or unwillingly accept the decisions of algorithms and Neural Network (NN) Models resulting in humans being decreasingly involved in the decision-making process. Human-machine cooperation and Human Autonomy Teaming (HAT) are an answer to this problem, joining the best of both sides into joined cognitive systems. Meaningful Human Control (MHC) is the concept of ensuring that humans have enough influence on the outcome of actions in HAT and hence stay morally responsible and legally accountable for their actions. This paper extends the concept of using an interaction mediator that facilitates the interaction between an automated and a human agent with interaction patterns to maintain MHC. The concept is applicable in almost any domain, and will be shown with the example of guiding a highly automated vehicle.

With increasing autonomous capabilities of machines the cooperation with humans becomes more and more important. In cooperative guidance and control, humans and machines either share or trade control in a cooperative way. Applied to vehicles, patterns of cooperative movements found in human-to-human interaction can be transferred to the cooperation and interaction of humans and highly automated vehicles. Starting with an overview of the concept of cooperative movement, the concept of interaction patterns is elaborated, on the one hand from an observation perspective and on the other hand from a design perspective. This includes image schemas, design patterns and interaction patterns.

With increasing autonomous capabilities and intelligence of machines and interconnectivity of humans and machines, interaction and cooperation become even more crucial for socio-cyber-physical systems. How can we organize the exploding number of design options in a way that it is easy for humans to understand and control these systems? Patterns for the design of interaction and cooperation could be the key for describing problems and generalizable solution patterns, which can be instantiated again in individual solutions. Modeling, matching and instantiating could be the fundamental use cases for those patterns. Understanding, designing, engineering and using are the essential activities in an interplay of researchers, designers, engineers and users of human-machine systems. A realistic dream is to have global databases of human-machine patterns and their effects so that the wheel does not have to be re-invented several times and common knowledge about patterns can be shared in our community.

Current complex socio-technical systems are in need of transparent communication between stakeholders and the technical system. Even more so, interaction becomes a key aspect to form cooperative systems like highly automated driving. In order to facilitate such interaction, common procedures and interaction patterns help to solve known repeating interaction problems. Interaction patterns have been extensively researched in recent years. They provide the benefit to standardize and describe an interaction activity between a human and a technical system. Previous forms of describing these interactions follow a somewhat open format, which increases the difficulty when reusing the proposed pattern, besides making the pattern’s representation at times too complex, which further hampers its reusability. This paper proposes a more flexible, semi-standardized approach to describe patterns for human machine cooperation. This method analyzes the interaction and groups the actions of both the human and the technical system into “interaction blocks”. These interaction blocks can then be re-arranged to adapt the interaction design to new applications or be reused and combined with other blocks to form new patterns. This paper sketches the concepts, and presents a couple of examples, which further demonstrate the benefit of using and designing an interaction with a pattern as a base.

With new technologies in the area of assistance and automation on the market, the dream of driving in automated vehicles becomes reality. While certain research areas predict fully autonomous vehicles in the near future, a more feasible composition still has a driver, at least as a back-up solution, operating the vehicle in certain situations with certain degrees of control, cooperating with the automation. In such complex highly automated human machine systems it is necessary to elucidate responsibilities of automation and human and assert controllability. To do so, a usable interaction design is required to support drivers in their driving tasks while maintaining the necessary controllability. This paper presents the concept of interaction patterns to improve usability and controllability when switching between different modes of automation. An interaction pattern implements image schemas to create intuitive interaction between a human and a computer. A study with highly automated truck convoys was carried out and the usability and controllability when switching between modes was evaluated.

Emerging technologies in the field of automatization meanwhile enable partially and highly automated vehicles in the aviation and automotive domains, where the human operators are assisted or (partially/temporarily) replaced in their tasks by advanced automation systems. Nevertheless due to technological limitations and ethical reasons, full autonomous vehicles in both domains might not be realizable in the near future. Instead system designs, which enable shared and cooperative guidance and control of vehicles by human operators and automation systems could be more feasible solutions. This paper sketches a common framework of shared and cooperative control that describes the two concepts not as different but as coinciding concepts for the shared intentionality, control and cooperation between humans and machines. A brief overview off developed shared and cooperative control designs in the aviation and ground vehicle domain is given.