Human and automation capabilities are the foundation of every human-autonomy interaction and interaction pattern. Therefore, machines need to understand the capacity and performance of human doing, and adapt their own behavior, accordingly. In this work, we address the concept of conjugated capabilities, i.e. capabilities that are dependent or interrelated and between which effort can be distributed. These may be used to overcome human limitations, by shifting effort from a deficient to a conjugated capability with performative resources. For example: A limited arm’s reach may be compensated by tilting the torso forward. We analyze the interrelation between elementary capabilities within the IMBA standard to uncover potential conjugation, and show evidence in data of post-rehabilitation patients. From the conjugated capabilities, within the example application of stationary manufacturing, we create a network of interrelations. With this graph, a manifold of potential uses is enabled. We showcase the graph’s usage in optimizing IMBA test design to accelerate data recordings, and discuss implications of conjugated capabilities on task allocation between the human and an autonomy.

Future crewed missions to the Moon will face significant environmental and operational challenges, posing risks to the safety and performance of astronauts navigating its inhospitable surface. Whilst head-up displays (HUDs) have proven effective in providing intuitive navigational support on Earth, the design of novel human-spaceflight solutions typically relies on costly and time-consuming analogue deployments, leaving the potential use of lunar HUDs largely under-explored. This paper explores an alternative approach by simulating navigational HUD concepts in a high-fidelity Virtual Reality (VR) representation of the lunar environment. In evaluating these concepts with astronauts and other aerospace experts (n=25), our mixed methods study demonstrates the efficacy of simulated analogues in facilitating rapid design assessments of early-stage HUD solutions. We illustrate this by elaborating key design challenges and guidelines for future lunar HUDs. In reflecting on the limitations of our approach, we propose directions for future design exploration of human-machine interfaces for the Moon.

Machines, e.g. empowered by AI and based on virtual models, can help to improve the quality of life. To exploit this potential and also integrate this with the real world and society, cooperation and teaming of these machines with humans, and with societies is crucial. Human-Machine Patterns can be a key concept to analyze, understand, design, engineer and evaluate the delicate interplay of humans and machines. Key issues here are to understand in which situations which agents should do, want to do, can do, will do and finally actually do which actions, and who then is accountable. This overview article is intended as an introduction into the special session on Shared and Cooperative Control, especially on patterns and models for controllability and resilience. It is a direct follow-up on the 2022's special session and overview article (which is also available on IEEE Xplore). It introduces the topic of shared and cooperative control of human-machine and human-AI systems, especially in the light of the new advances in AI technology. This paper gives a short overview on the state of research on interaction patterns, controllability and resilience, before it focuses on the fundamental aspects of which actor should do, wants to do, can do, does and will be accountable for a pattern, sub-pattern or action within a pattern. Examples of what can be achieved with this basic architecture are given, e.g. for the recognition of intent or for the support by assistant systems, using the automotive domain as a first application example.

The implicit assumption that human and autonomous agents have certain capabilities is omnipresent in modern teaming concepts. However, none formalize these capabilities in a flexible and quantifiable way. In this paper, we propose Capability Deltas, which establish a quantifiable source to craft autonomous assistance systems in which one agent takes the leader and the other the supporter role. We deduct the quantification of human capabilities based on an established assessment and documentation procedure from occupational inclusion of people with disabilities. This allows us to quantify the delta, or gap, between a team's current capability and a requirement established by a work process. The concept is then extended to the multidimensional capability space, which then allows to formalize compensation behavior and assess required actions by the autonomous agent.

To reach intuitive control of partially and highly automated machines, a smooth cooperation between both, human and machine, is necessary. A natural way to design cooperation is to use the structures of mental models already established in our minds by human-human or human-animal cooperation. These cooperation designs follow certain patterns to provide solutions in a variety of scenarios. By using interaction patterns, we form and trigger mental models in the human mind, which already uses patterns in a similar way. This paper provides a description of the structure of cooperation patterns. An exemplary pattern for the use case of control transition between a human and a driving automation is modelled based on data of an experiment of human drivers reacting to takeover requests. To use the patterns in human-machine cooperation systems, the novel concept of a pattern handler is introduced. The pattern handler can be instantiated in software and hardware, and matches the human behavior, automation actions, and environment data to cooperation patterns. Within the design phase, it helps with identifying and correcting flaws within the design.

Future automated driving systems (ADS) are discussed as having the ability to “override” driver control inputs. Yet, little is known about how drivers respond to this, nor how a human-machine interaction (HMI) for them should be designed. This work identifies intervention types associated with an ADS that has change control authority and outlines an experiment method which simulates a deficit in driver situation awareness, enabling the study of their responses to interventions in a controlled environment. In a simulator study (N = 18), it was found that drivers express more negative valence when their control input is blocked (p = .046) than when it is taken away. In safety-critical scenarios, drivers respond more positively to interventions (p = .021) and are willing to give the automation more control (p = .018). An experimental method and HMI design insights are presented and ethical questions about the development of automated driving are provoked.

The current advancement of technology with not only physical, but increasing cognitive functions widely coined as Artificial Intelligence (AI) leads to multiple situations where humans willingly or unwillingly accept the decisions of algorithms and Neural Network (NN) Models resulting in humans being decreasingly involved in the decision-making process. Human-machine cooperation and Human Autonomy Teaming (HAT) are an answer to this problem, joining the best of both sides into joined cognitive systems. Meaningful Human Control (MHC) is the concept of ensuring that humans have enough influence on the outcome of actions in HAT and hence stay morally responsible and legally accountable for their actions. This paper extends the concept of using an interaction mediator that facilitates the interaction between an automated and a human agent with interaction patterns to maintain MHC. The concept is applicable in almost any domain, and will be shown with the example of guiding a highly automated vehicle.

With increasing autonomous capabilities of machines the cooperation with humans becomes more and more important. In cooperative guidance and control, humans and machines either share or trade control in a cooperative way. Applied to vehicles, patterns of cooperative movements found in human-to-human interaction can be transferred to the cooperation and interaction of humans and highly automated vehicles. Starting with an overview of the concept of cooperative movement, the concept of interaction patterns is elaborated, on the one hand from an observation perspective and on the other hand from a design perspective. This includes image schemas, design patterns and interaction patterns.

With increasing autonomous capabilities and intelligence of machines and interconnectivity of humans and machines, interaction and cooperation become even more crucial for socio-cyber-physical systems. How can we organize the exploding number of design options in a way that it is easy for humans to understand and control these systems? Patterns for the design of interaction and cooperation could be the key for describing problems and generalizable solution patterns, which can be instantiated again in individual solutions. Modeling, matching and instantiating could be the fundamental use cases for those patterns. Understanding, designing, engineering and using are the essential activities in an interplay of researchers, designers, engineers and users of human-machine systems. A realistic dream is to have global databases of human-machine patterns and their effects so that the wheel does not have to be re-invented several times and common knowledge about patterns can be shared in our community.

Current complex socio-technical systems are in need of transparent communication between stakeholders and the technical system. Even more so, interaction becomes a key aspect to form cooperative systems like highly automated driving. In order to facilitate such interaction, common procedures and interaction patterns help to solve known repeating interaction problems. Interaction patterns have been extensively researched in recent years. They provide the benefit to standardize and describe an interaction activity between a human and a technical system. Previous forms of describing these interactions follow a somewhat open format, which increases the difficulty when reusing the proposed pattern, besides making the pattern’s representation at times too complex, which further hampers its reusability. This paper proposes a more flexible, semi-standardized approach to describe patterns for human machine cooperation. This method analyzes the interaction and groups the actions of both the human and the technical system into “interaction blocks”. These interaction blocks can then be re-arranged to adapt the interaction design to new applications or be reused and combined with other blocks to form new patterns. This paper sketches the concepts, and presents a couple of examples, which further demonstrate the benefit of using and designing an interaction with a pattern as a base.

While full automation of road vehicles remains a future goal, shared-control and semiautonomous driving-involving transitions of control between the human and the machine-are more feasible objectives in the near term. These alternative driving modes will benefit from new research toward novel steering control devices, more suitably where machine intelligence only partially controls the vehicle. In this article, it is proposed that when the human shares the control of a vehicle with an autonomous or semiautonomous system, a force control, or nondisplacement steering wheel (i.e., a steering wheel which does not rotate but detects the applied torque by the human driver) can be advantageous under certain schemes: tight rein or loose rein modes according to the H -metaphor. We support this proposition with the first experiments to the best of our knowledge, in which human participants drove in a simulated road scene with a force control steering wheel (FCSW). The experiments exhibited that humans can adapt promptly to force control steering and are able to control the vehicle smoothly. Different transfer functions are tested, which translate the applied torque at the FCSW to the steering angle at the wheels of the vehicle; it is shown that fractional order transfer functions increment steering stability and control accuracy when using a force control device. The transition of control experiments is also performed with both: a conventional and an FCSW. This prototypical steering system can be realized via steer-by-wire controls, which are already incorporated in commercially available vehicles.

The eight papers in this special issue focus on agent and system transparency in human-machine systems. The concept of transparency is investigated in a variety of contexts of human agent interaction—from a single robot to multiple heterogeneous agents and swarms. Two studies examine the effects of individual and cultural differences and, based on the compelling results, provide design recommendations related to transparent interfaces. One of the papers provides a thorough review on theoretical aspects of agent transparency and empirical findings on operator performance, situation awareness, trust and workload, among other outcomes. These measures are also among some of the most common metrics reported in studies in this special issue.

With new technologies in the area of assistance and automation on the market, the dream of driving in automated vehicles becomes reality. While certain research areas predict fully autonomous vehicles in the near future, a more feasible composition still has a driver, at least as a back-up solution, operating the vehicle in certain situations with certain degrees of control, cooperating with the automation. In such complex highly automated human machine systems it is necessary to elucidate responsibilities of automation and human and assert controllability. To do so, a usable interaction design is required to support drivers in their driving tasks while maintaining the necessary controllability. This paper presents the concept of interaction patterns to improve usability and controllability when switching between different modes of automation. An interaction pattern implements image schemas to create intuitive interaction between a human and a computer. A study with highly automated truck convoys was carried out and the usability and controllability when switching between modes was evaluated.

Emerging technologies in the field of automatization meanwhile enable partially and highly automated vehicles in the aviation and automotive domains, where the human operators are assisted or (partially/temporarily) replaced in their tasks by advanced automation systems. Nevertheless due to technological limitations and ethical reasons, full autonomous vehicles in both domains might not be realizable in the near future. Instead system designs, which enable shared and cooperative guidance and control of vehicles by human operators and automation systems could be more feasible solutions. This paper sketches a common framework of shared and cooperative control that describes the two concepts not as different but as coinciding concepts for the shared intentionality, control and cooperation between humans and machines. A brief overview off developed shared and cooperative control designs in the aviation and ground vehicle domain is given.

User state assessment in adaptive human-computer-interaction has often been equaled to the assessment of workload. More recently, approaches have surfaced that also focus on other state dimensions such as fatigue, situation awareness, and the emotional state. However, interrelations and dependencies between these state dimensions are not considered if each is assessed in isolation. Furthermore, it is often neglected that individual factors, factors of the work environment, and self-regulation-strategies of the operator strongly influence these state dimensions. We suppose that adaptation based on user state assessment is more successful if this set of interactions is integrated into a more holistic assessment approach. We therefore propose a model that aims to provide a more holistic view on user state assessment. The model also aims to be generic in the sense that it can be applied to diverse operational conditions. Application is illustrated by a recent aviation incident.

The increasing traffic volume confronts the road user with a challenging task. The high number of traffic deaths might not be reducible with passive safety alone. However systems that actively influence the guidance of vehicles, like assistance and automation systems, can make a difference towards higher safety, comfort and efficiency. Some of these systems completely take over single subtasks like speed or distance control. This, in turn can lead to effects like "out of the loop", where the driver withdraws from the actual task and even stops monitoring. In order to realize a safe automation system, the project H-Mode follows an approach where both, driver and assistance system are simultaneously affecting the vehicle, whereby the operator is kept in the loop and active. Moreover a haptic-multimodal communication between driver and automation is established by using active interfaces. Regarding this communication alternative control elements, especially two dimensional ones have to be considered.

In this paper, the theater-system technique, a method for agile designing and testing of system behavior and interaction concepts is described. The technique is based on the Wizard-of-Oz approach, originally used for emulating automated speech recognition, and is extended towards an interactive, user-centered design technique. The paper describes the design process using the theater-system technique, the technical build-up of the theater-system, and an application of the technique: the design of a haptic-multimodal interaction strategy for highly automated vehicles. The use of the theater-system in the design process is manifold: It is used for the concrete design work of the design team, for the assessment of user expectations as well as for early usability assessments, extending the principles of user-centered design towards a dynamically balanced design.