In our contribution, we investigate the impact of different modalities for tasking unmanned vehicles in Manned-Unmanned Teaming (MUM-T) scenarios. As pilots have to manage unmanned assets from the cockpit, human-machine interaction becomes critical to mission success. In this study we assessed the touch and voice modalities in a helicopter simulator, measuring workload, usability, and mission efficiency. It has been shown that voice interaction reduces workload and improves usability, as well as mission performance, while touch input remains valuable as a backup. The findings underline the need for improved interaction design in future MUM-T systems to enhance safety and mission efficiency in high-demand flight environments.

This study examines the impact of different tasking modalities - touch versus voice - on pilot performance in manned-unmanned teaming (MUM-T) mission scenarios. MUM-T operations place high cognitive and operational demands on military helicopter pilots, requiring simultaneous control of their own aircraft and coordination of unmanned aerial vehicles (UAVs). A simulator study was conducted analyzing subjective workload, head-down time, pilot flight behavior, and autopilot usage. Although pilots subjectively reported that voice tasking reduced head-down time and supported better flight performance, objective measurements showed only minor improvements or inconsistent results. Over 75 % head-down time was recorded across all conditions. Voice input alone does not notably mitigate visual demands. Task complexity, training, and context play a critical role. The findings underline the need for improved interaction design and more reliable flight automation in future MUM-T systems to enhance safety and mission efficiency in high-demand flight environments.

This study examines the effect of adaptive assistance in online mission planning for military aircraft pilots in Manned-Unmanned-Teaming (MUM-T) missions. We evaluated two adaptive approaches to select from three levels of assistance in a cockpit simulator experiment. Both approaches used the criticality of the situation to choose an assistance level, with one approach additionally integrating an eye-tracking-based measure of Situational Awareness (SA). Based on these triggers, the adaptive system used different levels of assistance for mission planning support. We compared these two adaptive conditions to a baseline setup without assistance and evaluated performance, workload, usability, and subjective SA. Performance was assessed by flight data and mission completion times, while workload, usability, and SA were assessed with standard questionnaires. Both assistance systems exceeded the non-assisted baseline in terms of mission time, workload management, SA, and usability. Pilots supported by adaptive assistance integrating both, criticality, and SA, showed improvements in subjective SA over the criticality-only-condition while demonstrating faster mission completion. The findings suggest that adaptive assistance, particularly when incorporating SA, can enhance pilot performance in MUM-T operations.

This paper examines the integration of cognitive mixed-initiative assistance via a Planning Assistance Agent in the context of Manned-Unmanned Teaming operations. The aim is to enhance mission planning, replanning, and execution in complex military air operations. The agent employs a hybrid Mixed-Initiative-Planning approach to autonomously adjust and optimize mission plans in real time, with the objective of reducing pilot workload, increasing situational awareness, and maintaining high mission success rates without increasing the risk of losses of unmanned assets. The agent integrates current environmental data and tactical situation changes into its planning processes, thereby closing the so-called “cognitive loop”. This is made possible by the use of sophisticated algorithms and planning problem modeling languages. The effectiveness of the Agent was evaluated through the participation of German Air Force pilots in both static and dynamic mission simulations. The dynamic simulations were conducted in a fully integrated Manned-Unmanned-Teaming fighter simulator, while the static missions required the pilots to create mission plans on a separate workstation. The simulations assessed the impact of the Agent on mission success and the pilot performance under varying levels of assistance. The results demonstrated that a situation-adapted assistance, which allows for dynamic and autonomous tactical adjustments by the Agent, most effectively enhances operational performance and pilot engagement without overwhelming the pilot or causing the pilot to over rely on automated systems.

Modern ultralight aircraft in general aviation are equipped with an onboard Pilot Assistance System (PAS) as a companion system, meant to guide the pilot in decision-making, e.g. with plan suggestions, especially in critical situations. For more meaningful guidance, the PAS must possess a continuous understanding of the context, i.e. the pilot’s intention, so that decision-making support is relevant. However, in realistic settings, the pilot’s intention is not communicated manually, but can only be proactively monitored by the PAS. This paper explores the possibility of embedding domain expertise using Hierarchical Task Network (HTN) planning to track the pilot’s intention, by recognising the pilot’s current goal task judging from the pilot’s actions. Furthermore, by leveraging probability theory for state estimation, we derive belief values to be associated with the recognised goal task, inferred from already executed actions which are in turn inferred from in-cockpit observable measurement data. Statistical evaluation using data collected from human-in-the-loop tests shows that our method for tracking the pilot’s intention is reliable enough to provide the PAS with a contextual understanding in real time.

Manned-Unmanned Teaming (MUM-T) is an ensemble of manned and unmanned vehicles operating as a team to achieve the same set of goals. Such teaming is highly beneficial, notably in overcoming the limits of direct communication link to the unmanned vehicles, as well as in enhancing capabilities of a manned vehicle by leveraging multiple accompanying unmanned vehicles. However, this can in times overstrain the command and control capacity of the human operator(s) on board of the manned vehicle(s), unless if the unmanned vehicles possess some “understanding” of the human operators’ decision-making behaviors, in which case, they can act like real “team players” to proactively support the manned vehicle, instead of waiting passively for successive commands. In this study, we investigate the possibility of learning decision-making behaviors of a human operator on board of a manned vehicle in charge of commanding multiple accompanying unmanned vehicles. We base our investigation on a rescue mission involving a manned helicopter and several unmanned aerial vehicles to collect training and validation data using software-in-the-loop simulations. By extracting meaningful features and by performing a clustering on the features on the training dataset, we validate and analyze the learned pattern of commanding the unmanned vehicles.

By nature, military Manned-Unmanned Teaming (MUM-T) missions are characterized by complex state spaces, uncertainty and multiple types/sources of information that the pilot needs to assess in a time critical manner to derive tactical decisions. In order to support the helicopter crew in such missions, we investigate a software agent that interprets available situational and mission related information to create machine situational awareness. In this article, we present a work in progress on the concept and implementation of a tactical situation assessment system using an Influence Map evaluation polytree for knowledge representation. Furthermore, we look at the problem from a human autonomy teaming perspective. The agent provides tactical recommendations to the pilot and shares its inferred knowledge about possible future critical states. We present the current state of integration in our helicopter research flight simulator and outline challenges concerning effective human-machine cooperation in tactical decision-making.

This study focuses on the transparent design of a cognitive agent to enhance situation awareness in two aspects of a human-agent teaming application: assisted system management and mixed-initiative mission planning. Adaptive and complex agent behavior might result in the failure to comprehend resulting interventions, a decrease in trust, and a loss of overall situation awareness. This study describes and validates a concept for transparent agent design by adopting the transparency strategies proposed by the “situation awareness-based agent transparency model.” The overall objective was to improve the human operator's perception, comprehension, and projection of the agent's support. The concept was applied to the prototype of a workload-adaptive cognitive agent, which supports a helicopter crew during mission planning and execution in complex and dynamically changing multi-vehicle missions. A human-in-the-loop experiment revealed enhancements in situation awareness and performance. Subjective trust measures implied an increase in human-like characteristics of the cognitive agent. The results and the potential for further research are discussed.

High-Altitude Pseudo-Satellites (HAPS) are long-endurance, fixed-wing, lightweight Unmanned Aerial Vehicles (UAVs) that operate in the stratosphere and offer a flexible alternative for ground activity monitoring/imaging at specific time windows. As their missions must be planned ahead (to let them operate in controlled airspace), this paper presents a Genetic Algorithm (GA)-guided Hierarchical Task Network (HTN)-based planner for multiple HAPS. The HTN allows to compute plans that conform with airspace regulations and operation protocols. The GA copes with the exponentially growing complexity (with the number of monitoring locations and involved HAPS) of the combinatorial problem to search for an optimal task decomposition (that considers the time-dependent mission requirements and the time-varying environment). Besides, the GA offers a flexible way to handle the problem constraints and optimization criteria: the former encodes the airspace regulations, while the latter measures the client satisfaction, the operation efficiency and the normalized expected mission reward (that considers the wind effects in the uncertainty of the arrival-times at the monitoring-locations). Finally, by integrating the GA into the HTN planner, the new approach efficiently finds overall good task decompositions, leading to satisfactory task plans that can be executed reliably (even in tough environments), as the results in the paper show.

The Institute of Flight Systems (IFS) at the Bundeswehr University Munich (BUM) conducts research in the area of military helicopter Manned-Unmanned Teaming (MUM-T) scenarios. In order to support the crews in suchlike missions, the IFS investigates functions that enhances automated mission management concerning the command of unmanned aerial vehicles (UAVs) during the execution of the mission. Previously conducted experiments and interviews with pilots of the German Armed Forces have shown that there is a need for the system to have a higher level of tactical situational awareness, enabling it to provide tactically relevant support. Typically, military scenarios are characterized by high complexity regarding the state space and information available. Influence Maps (IM) pose a way to represent and manage tactical information of these complex environments. In this article, we show how information of an operating area can be represented using IMs and how these IMs can then be used to conduct tactical inference or assessment. The resulting IMs are used to assess the helicopter flight route and generate tactical reconnaissance target candidates.

Navigating in a vector field is a challenging problem for many autonomous vehicles. This article focuses on a High-Altitude Pseudo-Satellite (HAPS) operating in a wind field with wind magnitude comparable to its airspeed. In addition to navigating from a start to a goal point, the HAPS is expected to carry out patrolling missions spanning long hours/ days, within which the physical environment (e.g. wind field, weather-critical no-go areas, fly zones etc.) vary, resulting in either an improvement or a deterioration of mission fulfillment. A time-dependent hybrid mission planning framework is proposed in this work which consists firstly of a hierarchical strategic planner that produces very quickly multiple sequences of tasks (or rather "premature plans") that are ranked with roughly estimated objective or penalty values, and secondly of a tactical planner that refines the values of each premature plan to help the operator assess the plans better. The framework was implemented on the long-term offline planner for HAPS in patrolling missions. Test results using an independent six degrees-of-freedoms HAPS simulator and historical weather data are provided and analyzed.

This article describes a method for predicting future mental states and workload of military helicopter crews, and how adaptive technical assistance is derived. A mission plan and a model of pilot tasks are the basis for predicting future task situations. Combined with knowledge of the mental resource demands of these task situations, workload peaks can be identified before they occur. A task-based, context-rich representation of the crews' mental state enables an adaptive associate system to support the crew, while preventing high-workload task situations. Therefore, the associate system changes the task sharing between the human operator and the automated system online by using different levels of automation and a restrained intervention strategy. This concept is implemented as software agent in a helicopter mission simulator and will be evaluated in pilot-in-the-loop experiments in the near future.

In this article we propose a concept of cognitive pilot assistance for on-board mission (re-)planning. Current research activities address the delegation of complex multi-vehicle planning tasks to automation, as well as the related monitoring tasks. However, opacity and loss of situation awareness may arise as a consequence of delegation. To counteract these shortcomings, we suggest a mixed-initiative planner approach. Adequate assistance shall be ensured by a planning strategy model, which represents the pilots' mental process during the planning. A pilot observer determines the pilot's current activities. In a next step, these activities are analyzed to identify the pilot's planning progress. The planning progress and other factors, for example situational changes, are used to shape the intervention policy. An overview of the current implementation status and the future work is given. The proposed concept is investigated in our military manned unmanned teaming application.

We investigate the operationalization of mental workload (MWL) for adaptive pilot support. Common approaches have proven their benefits either for functional allocation in automation and crew station design, or in statistically grounded human-machine system evaluations. In this article, we argue why many established methods are not viable for a continuous, nonintrusive, context-rich, absolute rating of MWL. Instead, we suggest a novel threefold concept. Firstly, MWL depends on the given work objectives and the human operator's tasks. Secondly, we understand MWL as induced by the current human activity and related mental resource demands. Thirdly, MWL influences task and activity related, observable human behavior patterns. We argue that each of these determinants increasingly refines the quantification of MWL. In this article, we present the current state of our concept and prototype implementation. The concept has consequences for the design of cognitive pilot associate systems in manned-unmanned systems in military aviation.

An artificial cognitive agent, guiding a UAV based on operator-given tasks, is introduced onboard a conventionally automated aircraft in order to support the human operator. The agent acts in two roles - as subordinate to the human and at the same time as supervisor to the conventional onboard automation. The hierarchical relationship is described by the term Agent Supervisory Control. The ACU performs Supervisory Functions, with which the tasks of the operator are aggregated into single actions. The agent considers automation functions of the UAV as capabilities that are managed and incorporated into its planning and execution process in accordance to their availability and requirements. Task delegation is highly adaptable and aims at situation adequate supervision for the operator. The agent uses its capability awareness in its subordinate role to correct possible mismatches in the mental model of the human.

Helicopter Emergency Medical Service (HEMS) missions are associated with high workload for the cockpit crew. Cognitive assistant systems investigated today to counteract high workload issues are proven beneficial in principle, but may also induce additional load for the pilot, especially if the system intervenes when the human operator has little or no free cognitive resources to adopt the offered support. The basis for counteracting such automation induced issues is the automatic, reliable, task-related assessment of the current workload of the human operator. In this article we present a concept and prototype implementation to estimate the usage of mental resources of the human pilot and his current workload level in HEMS missions. Furthermore we describe first evaluation experiments conducted in our research helicopter mission simulator.

In this article, we present the architectural and algorithmic details for a COgnitive System Architecture that uses a Centralized Ontology with Specific Algorithms (COSA2). COSA2is a layered intelligent agent framework on the basis of the modified Rasmussen model of human performance. It encompasses integrated algorithmic support for goal-oriented situation interpretation, dynamic planning and plan execution, as well as provisions for reactive behavior. A unique feature is the claim for an expressive, centralized knowledge representation, used by all functions to ensure consistency. The framework is being applied to different problems in the domain of uninhabited aerial vehicles. This article focuses on high level, concept-based behavior and illustrates the modeling and processing details by a simplified UAV mission management example.