This position paper argues that the use of generative artificial intelligence (GenAI) to control, direct, guide or govern any weapon, either in situ or remotely, should be prohibited by government agencies and non-governmental organizations. Such a moratorium should exist until hallucinations can be successfully modeled and predicted. Generative AI is inherently unreliable and not appropriate in environments that could result in the loss of life.

Many industrial image segmentation systems require training on large annotated datasets, but there is little standardization for producing training annotations. Data is often obtained via crowdsourcing, but many labeling task configurations are set by convenience and may have unintended effects on data quality. In this work, we (1) demonstrate a new software tool for running crowdsourced image segmentation experiments, (2) present a dataset capturing variation in segmentation annotations produced under different task configurations, and (3) experimentally evaluate the quality of the annotations produced by these different configurations. We show annotation quality can be significantly improved by paying annotators per annotated object rather than per image, and by leveraging paintbrush-style drawing tools rather than polygon or curve drag tools. We also show that some task complexity is required to maintain annotator engagement and sufficient task performance. Finally, we show that many configuration-related annotation errors degrade model training performance, but that models can tolerate error error patterns that are common across crowdsourced annotation schemes.

The need to design for appropriate human-autonomy teaming has become increasingly important as systems grow in complexity, especially those that require time-pressured interactions like in unmanned aerial vehicle (UAV) operations. However, it is not always clear whether operators develop effective strategies for computer-based technologies. When operators are given such tools, their performances can be statistically compared but often such assessments only provide summative information. These comparisons do not indicate how and why technology influenced people's strategies and actions. To fill this gap, we utilized Hidden Markov Models (HMMs) to represent strategies employed by operators in first-person control of UAVs for inspection tasks. The resulting models captured differences in strategies for people who both succeeded and crashed, as well as those who were overconfident in their self-assessments, and those who were not. People who were not overconfident exhibited less risky strategies and were more successful. These findings were further strengthened by a quantitative state similarity metric, which indicated where and for who possible interventions could improve outcomes. This application of HMMs to operator strategy representation could help to identify effective operator strategy development in the use of computer-based technologies, and what kind of interventions could be the most effective in improving outcomes.

Advanced Driving Assist Systems (ADAS) are on the rise in new cars, including versions that embed artificial intelligence in computer vision systems that leverage deep learning algorithms. Because these systems, at the present time, cannot operate in all operational driving domains, they employ some type of driver monitoring system for assessing driver attention, so that drivers can effectively take control if and when an ADAS system can no longer control the car. To determine the reliability of a driver alerting system when linked to autonomy that leverages deep learning, a set of increasingly complex tests were conducted on three Tesla Model 3 vehicles. Tests were conducted on a highway and a closed test track to test road departure and construction zone detection capabilities. Results revealed significant between- and within-vehicle variation on a number of metrics related to driver monitoring, alerting, and safe operation of the underlying autonomy. In some cases, cars performed better than expected but all cars exhibited both inconsistent and unsafe behaviors as well as poor driver alerting. These results highlight that a post-deployment regulatory process is ill-equipped to flag significant issues in vehicles with embedded artificial intelligence.

Human-automation collaborations, like automated driving assistance and piloting drones, have become prevalent as these technologies become more commonplace. Designers need tools that help them understand how and why design interventions may change the strategies of operators in such complex human supervisory control systems. To this end, we demonstrate that when the divergence metric is applied to Hidden Markov Model (HMM) comparisons, it can accurately capture statistical differences between operator strategies for interfaces that embody different tasks. However, the use of such an approach is problematic when used to compare HMM strategy models with non-equivalent observations. To address this limitation, we developed an observation reduction approach and conducted a sensitivity analysis to assess the impact of this approach. Our results show that when comparing two non-equivalent interfaces, our observation reduction approach does not fundamentally change the divergence metric, thus allowing for direct model comparison. The results further show that HMMs from different interfaces produce a much higher divergence metric than model comparison from the same people who repeatedly use the same interface. Future work will examine if this method can detect differences in models with different tasks or modified interfaces.

In this work, we synthesize collaboration protocols for human-unmanned aerial vehicle (H-UAV) command and control systems, where the human operator aids in securing the UAV by intermittently performing geolocation tasks to confirm its reported location. We first present a stochastic game-based model for the system that accounts for both the operator and an adversary capable of launching stealthy false-data injection attacks, causing the UAV to deviate from its path. We also describe a synthesis challenge due to the UAV's hidden-information constraint. Next, we perform human experiments using a developed RESCHU-SA testbed to recognize the geolocation strategies that operators adopt. Furthermore, we deploy machine learning techniques on the collected experimental data to predict the correctness of a geolocation task at a given location based on its geographical features. By representing the model as a delayed-action game and formalizing the system objectives, we utilize off-the-shelf model checkers to synthesize protocols for the human-UAV coalition that satisfy these objectives. Finally, we demonstrate the usefulness of the H-UAV protocol synthesis through a case study where the protocols are experimentally analyzed and further evaluated by human operators.

Human supervisory control (HSC) is a widely used knowledge-based control scheme, in which human operators are in charge of planning and making high-level decisions for systems with embedded autonomy. With the variability of operators' behaviors in such systems, the stability of an operator modeling technique, i.e., that a modeling approach produces similar results across repeated applications, is critical to the extensibility and utility of such a model. Using an unmanned vehicle simulation testbed where such vehicles can be hacked, we compared two operator behavioral models from two different experiments using a hidden Markov modeling (HMM) approach. The resulting HMM models revealed operators' dominant strategies when conducting hacking detection tasks. The similarity between these two models was measured via multiple aspects, including model structure, state distribution, divergence distance, and co-emission probability distance. The similarity measure results demonstrate the stability of modeling human operators in HSC scenarios using HMM models. These results indicate that even when operators perform differently on specific tasks, such an approach can reliably detect whether strategies change across different experiments.

As new technologies are introduced into rail operations, models are needed to represent the task load of operators to identify periods of extreme workload that could be mitigated through technological interventions. To this end, a computational model is described to quantitatively simulate freight rail operator workload to understand the impacts of inserting intelligent automation on different crew configurations. A detailed task analysis served as the basis for identifying tasks performed during transit. Utilizing task characteristics and operating conditions as inputs, a discrete event simulation was designed to predict human operator workload. Results show that during heavy-traffic conditions, the presence of automation can impact the locomotive engineer performance more than the presence of a freight conductor in a short-haul freight rail setting. However, under typical conditions, assistance may not be as beneficial for human operator performance.

An increasingly relevant security issue for unmanned aerial vehicles (UAVs, also known as drones) is the possibility of a global positioning system (GPS) spoofing attack. Given the existing problems in current GPS spoofing detection techniques and human visual advantages in searching and localizing targets, we propose a human-autonomy collaborative approach of human geolocation to assist UAV control systems in detecting GPS spoofing attacks. An interactive testbed and experiment were designed and used to evaluate this approach, which demonstrated that human-autonomy collaborative hacking detection is a viable concept. Using the hidden Markov model (HMM) approach, operator behavior patterns and strategies from the experiment were modeled via hidden states and transitions among them. These models revealed two dominant hacking detection strategies. Statistical results and expert performer evaluations show no significant difference between different hacking detection strategies in terms of correct detection. The detection strategy model suggests areas of future research in decision support tool design for UAV hacking detection. Also, the development of HMMs presents the feasibility of quantitatively investigating operator behavior patterns and strategies in human supervisory control scenarios.

Unmanned Aerial Vehicles (UAVs), also known as drones, have extensive applications in civilian rescue and military surveillance realms. A common drone control scheme among such applications is human supervisory control, in which human operators remotely navigate drones and direct them to conduct high-level tasks. However, different levels of autonomy in the control system and different operator training processes may affect operators’ performance in task success rate and efficiency. An experiment was designed and conducted to investigate such potential impacts. The results showed us that a dedicated supervisory drone control interface tended toward increased operator successful task completion as compared to an enhanced teleoperation control interface, although this difference was not statistically significant. In addition, using Hidden Markov Models, operator behavior models were developed to further study the impact of operators’ drone control strategies as a function of differing levels of autonomy. These models revealed that people with both supervisory and enhanced teleoperation control training were not able to determine the right control action at the right time to the same degree that people with just training in the supervisory control mode. Future work is needed to determine how trust plays a role in such settings.

Recent advances in unmanned and autonomous vehicle technology are accelerating the push to integrate these vehicles into environments, such as the National Airspace System, the national highway system, and many manufacturing environments. These environments will require close collaboration between humans and vehicles, and their large scales mean that real-world field trials may be difficult to execute due to concerns of cost, availability, and technological maturity. This paper describes the use of an agent-based model to explore the system-level effects of unmanned vehicle implementation on the performance of these collaborative human-vehicle environments. In particular, this paper explores the introduction of three different unmanned vehicle control architectures into aircraft carrier flight deck operations. The different control architectures are tested under an example mission scenario using 22 aircraft. Results show that certain control architectures can improve the rate of launches, but these improvements are limited by the structure of flight deck operations and nature of the launch task (which is defined independently of vehicles). Until the launch task is improved, the effects of unmanned vehicle control architectures on flight deck performance will be limited.

With improving automation in many critical domains, operators will be expected to handle long periods of low task load while monitoring a system, and possibly responding to emergent situations. Monitoring the psychophysiological state of the operator during low task load may detect maladapted attention states in order to predict performance and facilitate a more effective workload transition during critical periods. This research explored the question of detecting anomalous attention states during transitions to high workload following extended periods of boredom using a non-invasive neuroimaging technique called functional near-infrared spectroscopy (fNIRS). Subjects at the point of lowest engagement and priming had a diminished hemodynamic response and performed worse on missile defense task, showing fNIRS may be useful for concurrent monitoring of the operator in such settings. RESEARCH HIGHLIGHTS Functional near-infrared spectroscopy brain sensing is feasible for use in long duration (3 h) tasks. Hemodynamic response was diminished during the middle of a long duration, low task load simulation when engagement and priming were lowest. fNIRS did not detect a change in workload, but did reflect temporal changes in event onset, which could be used to automatically adapt a system when an operator is in a degraded attention state.

Planning operations across a number of domains can be considered as resource allocation problems with timing constraints. An unexplored instance of such a problem domain is the aircraft carrier flight deck, where, in current operations, replanning is done without the aid of any computerized decision support. Rather, veteran operators employ a set of experience-based heuristics to quickly generate new operating schedules. These expert user heuristics are neither codified nor evaluated by the United States Navy; they have grown solely from the convergent experiences of supervisory staff. As unmanned aerial vehicles (UAVs) are introduced in the aircraft carrier domain, these heuristics may require alterations due to differing capabilities. The inclusion of UAVs also allows for new opportunities for on-line planning and control, providing an alternative to the current heuristic-based replanning methodology. To investigate these issues formally, we have developed a decision support system for flight deck operations that utilizes a conventional integer linear program-based planning algorithm. In this system, a human operator sets both the goals and constraints for the algorithm, which then returns a proposed schedule for operator approval. As a part of validating this system, the performance of this collaborative human-automation planner was compared with that of the expert user heuristics over a set of test scenarios. The resulting analysis shows that human heuristics often outperform the plans produced by an optimization algorithm, but are also often more conservative.

There has recently been a significant amount of activity in developing supervisory control algorithms for multiple unmanned aerial vehicle operation by a single operator. While previous work has demonstrated the favorable impacts that arise in the introduction of increasingly sophisticated autonomy algorithms, little work has performed an explicit comparison of different types of multiple unmanned vehicle control architectures on operator performance and workload. This paper compares a vehicle-based paradigm (where a single operator individually assigns tasks to unmanned assets) to a task-based paradigm (where the operator generates a task list, which is then given to the group of vehicles that determine how to best divide the tasks among themselves.) The results demonstrate significant advantages in using a task-based paradigm for both overall performance and robustness to increased workload. This effort also demonstrated that while previous video gaming experience mattered for performance, the degree of experience that demonstrated benefit was minimal. Further work should focus on designing a flexible automated system that allows operators to focus on a primary goal, but also facilitate lower level control when needed without degradation in performance.

Simultaneously controlling multiple robots requires multiple operators working together as a team. Determining how to construct the team to promote performance and reduce workload are critical questions that must be answered in these settings. To this end, we investigated the effect of team structure and scheduling notification on operators' performance, subjective workload, work processes, and communication using a human-in-the-loop experiment. In an urban search and rescue setting, we compared a pooled condition, in which team members shared control of 24 robots, with a sector condition, in which each team member controlled half of all the robots. For scheduling notification, an alert was given when the operator spent too much time on one robot and either suggested or forced the operator to change to another robot. A discrete-event simulation model was constructed to model the teamwork in supervisory control of multiple robots. The model was significantly improved by the inclusion of a behavior termed as “backup.” Backup behavior is a critical coordination mechanism often observed in teams, but rarely explicitly modeled. Pooled teams showed an advantage when performing backup behaviors in both the experiment and the model. However, other factors must be considered when making a decision on what team structure to use.

There is an increasing requirement for advanced analytical methodologies to help military intelligence analysts cope with the growing amount of data they are saturated with on a daily basis. Specifically, within the context of terror network analysis, one of the largest problems is the transformation of raw tabular data into a visualization that is easily and effectively exploited by intelligence analysts. Currently, the primary method within the intelligence do-main is the node-link visualization, which encodes data sets by depicting the ties between nodes as lines between objects in a plane. This method, although useful, has limitations when the size and complexity of data grows. The matrix offers an alternate perspective because the two dimensions of the matrix are arrayed as an actors x actors matrix. This paper describes an experiment investigating node-link and matrix visualization techniques within social network analysis, and their effectiveness for the intelligence tasks of: 1) identifying leaders and 2) identifying clusters. The sixty participants in the experiment were all Air Force intelligence analysts and we provide recommendations for building visualization tools for this specialized group of users.

Operators currently controlling unmanned aerial vehicles report significant boredom, and such systems will likely become more automated in the future. Similar problems are found in process control, commercial aviation and medical settings. To examine the effect of boredom in such settings, a long-duration low-task-load experiment was conducted. Three low-task-load levels requiring operator input every 10, 20 or 30 min were tested in a 4-h study, using a multiple unmanned vehicle simulation environment that leverages decentralized algorithms for sometimes-imperfect vehicle scheduling. Reaction times to system-generated events generally decreased across the 4 h, as did participants' ability to maintain directed attention. Overall, the participants spent almost half of the time in a distracted state. The top performer spent the majority of time in directed and divided attention states. Unexpectedly, the second-best participant, only 1% worse than the top performer, was distracted for almost one-third of the experiment, but exhibited a periodic switching strategy, allowing himself to pay just enough attention to assist the automation when needed. Indeed, four of the five top performers were distracted for more than one-third of the time. These findings suggest that distraction due to boring, low-task-load environments can be effectively managed through efficient attention switching. Future work is needed to determine optimal frequency and duration of attention state switches, given various exogenous attributes, as well as individual variability. These findings have implications for the design of and personnel selection for supervisory control systems where operators monitor highly automated systems for long durations with only occasional or rare input.

Deciding if and what objects should be engaged in a ballistic missile defense system (BMDS) scenario involves a number of complex issues. The system is large, and the timelines may be on the order of a few minutes, which drives designers to automate these systems. The critical nature of ballistic missile defense engagement decisions however, suggests exploring a human-in-the-loop approach to allow for judgment, knowledge-based decisions, and the ability to override automation decisions. This BMDS problem is reflective of the function allocation conundrum faced in many supervisory control systems, which is how to determine which functions should be mutually exclusive and which should be collaborative between humans and automation. This paper motivates and outlines two experiments that quantitatively investigated human/automation tradeoffs in the specific domain of tracking problems. Participants in both experiments were tested in their ability to smooth trajectories in different scenarios. In the first experiment, they clearly demonstrated an ability to assist an algorithm in more difficult, shorter timeline scenarios. The second experiment combined the strengths of both human and automation in order to produce a collaborative effort. Comparison of the collaborative effort to the algorithm showed that adjusting the criterion for having human participation could significantly improve solutions. Future work should focus on further examination of appropriate criteria.

Simultaneously controlling increasing numbers of robots requires multiple operators working together as a team. Helping operators allocate attention among different robots and determining how to construct the human-robot team to promote performance and reduce workload are critical questions that must be answered in these settings. To this end, we investigated the effect of team structure and search guidance on operators' performance, subjective workload, work processes and communication. To investigate team structure in an urban search and rescue setting, we compared a pooled condition, in which team members shared control of 24 robots, with a sector condition, in which each team member control half of all the robots. For search guidance, a notification was given when the operator spent too much time on one robot and either suggested or forced the operator to change to another robot. A total of 48 participants completed the experiment with two persons forming one team. The results demonstrate that automated search guidance neither increased nor decreased performance. However, suggested search guidance decreased average task completion time in Sector teams. Search guidance also influenced operators' teleoperation behaviors. For team structure, pooled teams experienced lower subjective workload than sector teams. Pooled teams communicated more than sector teams, but sector teams teleoperated more than pool teams.

For future systems that require one or a small team of operators to supervise a network of automated agents, automated planners are critical since they are faster than humans for path planning and resource allocation in multivariate, dynamic, time-pressured environments. However, such planners can be brittle and unable to respond to emergent events. Human operators can aid such systems by bringing their knowledge-based reasoning and experience to bear. Given a decentralized task planner and a goal-based operator interface for a network of unmanned vehicles in a search, track, and neutralize mission, we demonstrate with a human-on-the-loop experiment that humans guiding these decentralized planners improved system performance by up to 50%. However, those tasks that required precise and rapid calculations were not significantly improved with human aid. Thus, there is a shared space in such complex missions for human-automation collaboration.

Unmanned aerial vehicles (UAVs) provide unprecedented access to imagery of possible ground targets of interest in real time. The availability of this imagery is expected to increase with envisaged future missions of one operator controlling multiple UAVs. This research investigates decision models that can be used to develop assistive decision support for UAV operators involved in these complex search missions. Previous human-in-the-loop experiments have shown that operator detection probabilities may decay with increased search time. Providing the operators with the ability to requeue difficult images with the option of relooking at targets later was hypothesized to help operators improve their search accuracy. However, it was not well understood how mission performance could be impacted by operators performing requeues with multiple UAVs. This work extends a queuing model of the human operator by developing a retrial queue model (ReQM) that mathematically describes the use of relooks. We use ReQM to generate performance predictions through discrete event simulation. We validate these predictions through a human-in-the-loop experiment that evaluates the impact of requeuing on a simulated multiple-UAV mission. Our results suggest that, while requeuing can improve detection accuracy and decrease mean search times, operators may need additional decision support to use relooks effectively.

This research, a collaboration between MIT and ABB/Ventyx, is focused on the development of a mobile interface for field workers in power repair settings and field service delivery. A Human Systems Engineering (HSE) approach of Plan, Analyze, and Design was utilized to develop the interface, which included a Hybrid Cognitive Task Analysis (hCTA) that identified requirements for the envisioned interface. This paper overviews the results of the HSE process and presents a preliminary design for the mobile interface that emerged during initial display prototyping.

In time-critical systems in which a human operator supervises multiple semiautomated tasks, failure of the operator to focus attention on high-priority tasks in a timely manner can lower the effectiveness of the system and potentially result in catastrophic consequences. These systems must integrate computer-based technologies that help the human operator place attention on the right tasks at the right times to be successful. One way to assist the operator in this process is to compute where the operator's attention should be focused and then use this computation to influence the operator's behavior. In this paper, we analyze the ability of a particular modeling method to make such computations for effective attention allocation in human-multiple-robot systems. Our results demonstrate that it is not sufficient to simply compute and dictate how operators should allocate their attention. Rather, in stochastic domains, where small changes in either the endogenous or exogenous environment can dramatically affect model fidelity, model predictions should guide rather than dictate operator attentional resources so that operators can effectively exercise their judgment and experience.

Discrete-event simulations for futuristic unmanned vehicle (UV) systems enable a cost- and time-effective methodology for evaluating various autonomy and human-automation design parameters. Operator mental workload is an important factor to consider in such models. We suggest that the effects of operator workload on system performance can be modeled in such a simulation environment through a quantitative relation between operator attention and utilization, i.e., operator busy time used as a surrogate real-time workload measure. To validate our model, a heterogeneous UV simulation experiment was conducted with 74 participants. Performance-based measures of attention switching delays were incorporated in the discrete-event simulation model by UV wait times due to operator attention inefficiencies (WTAIs). Experimental results showed that WTAI is significantly associated with operator utilization (UT) such that high UT levels correspond to higher wait times. The inclusion of this empirical UT-WTAI relation in the discrete-event simulation model of multiple UV supervisory control resulted in more accurate replications of data, as well as more accurate predictions for alternative UV team structures. These results have implications for the design of future human-UV systems, as well as more general multiple task supervisory control models.

In the future vision of allowing a single operator to remotely control multiple unmanned vehicles, it is not well understood what cognitive constraints limit the number of vehicles and related tasks that a single operator can manage. This paper illustrates that, when predicting the number of unmanned aerial vehicles (UAVs) that a single operator can control, it is important to model the sources of wait times (WTs) caused by human-vehicle interaction, particularly since these times could potentially lead to a system failure. Specifically, these sources of vehicle WTs include cognitive reorientation and interaction WT (WTI), queues for multiple-vehicle interactions, and loss of situation awareness (SA) WTs. When WTs were included, predictions using a multiple homogeneous and independent UAV simulation dropped by up to 67%, with a loss of SA as the primary source of WT delays. Moreover, this paper demonstrated that even in a highly automated management-by-exception system, which should alleviate queuing and WTIs, operator capacity is still affected by the SA WT, causing a 36% decrease over the capacity model with no WT included.

Efforts are underway to make it possible for a single operator to effectively control multiple robots. In these high workload situations, many questions arise including how many robots should be in the team (Fan-out), what level of autonomy should the robots have, and when should this level of autonomy change (i.e., dynamic autonomy). We propose that a set of metric classes should be identified that can adequately answer these questions. Toward this end, we present a potential set of metric classes for human-robot teams consisting of a single human operator and multiple robots. To test the usefulness and appropriateness of this set of metric classes, we conducted a user study with simulated robots. Using the data obtained from this study, we explore the ability of this set of metric classes to answer these questions.

In recent years, much research has focused on making possible single-operator control of multiple robots. In these high workload situations, many questions arise including how many robots should be in the team, which autonomy levels should they employ, and when should these autonomy levels change? To answer these questions, sets of metric classes should be identified that capture these aspects of the human-robot team. Such a set of metric classes should have three properties. First, it should contain the key performance parameters of the system. Second, it should identify the limitations of the agents in the system. Third, it should have predictive power. In this paper, we decompose a human-robot team consisting of a single human and multiple robots in an effort to identify such a set of metric classes. We assess the ability of this set of metric classes to: 1) predict the number of robots that should be in the team and 2) predict system effectiveness. We do so by comparing predictions with actual data from a user study, which is also described.

ABET engineering criteria (EC) 2000 specify that engineering colleges and universities must ensure all students understand their professional and ethical responsibilities upon graduation. This paper discusses the strategies used in developing and teaching an online engineering ethics class developed in part to meet this requirement. Web-based instruction can provide the opportunity to bridge both distance and cultural gaps, which makes it uniquely suited to connect engineering ethics communities both in the United States as well as globally. In addition, the effectiveness of this Web-based approach in teaching ethics is discussed as well as how this approach can be extended to include international student representation.

Designing decision support systems for revolutionary domains is problematic because approaches such as cognitive task and work analyses are difficult to implement due to the lack of established domains or users. An example of such a conundrum is the development of in-flight retargeting capabilities for the US Navy's Tomahawk missile. The current version is a "fire-and-forget" missile. However, a new upgrade will allow redirection of missiles in-flight. This new capability means that not only will battlefield commanders have more flexibility, but also that an entire system of human control will be needed where none previously existed. For decision support systems in revolutionary domains like that of the new Tomahawk, the lack of an existing domain makes it difficult to design an effective decision support system. Using the Tomahawk as a case study, this paper demonstrates how the cognitive work analysis can be modified for application to intentional revolutionary domains.