Multi-agent systems and agent-oriented methodologies support analysis, characterization and development of complex software systems. These methodologies introduce different definitions for the essential components of multi-agent systems and cover different phases of the system development life cycle. Therefore, appropriate frameworks for evaluation and comparison of different methodologies would support developers to adopt the best methodology, or a combination of different methodologies, based on the project requirements. This review covers the system development phases and the main conceptual components in the context of multi-agent systems. Then, the evaluation frameworks proposed in the literature for comparison and evaluation of agent-oriented methodologies are reviewed. Evaluation frameworks proposed in the literature are categorized into three categories: methodology-based, phase-based and feature-based evaluation frameworks. The paper concludes with the agent-oriented methodologies’ usage challenges, their current limitations and potential future directions.

OBJECTIVE: This study introduces a temporal condition pattern mining methodology to address the sparse nature of coded condition concept utilization in electronic health record data. As a validation study, we applied this method to reveal condition patterns surrounding an initial diagnosis of pediatric asthma. MATERIALS AND METHODS: The SPADE (Sequential PAttern Discovery using Equivalence classes) algorithm was used to identify common temporal condition patterns surrounding the initial diagnosis of pediatric asthma in a study population of 71 824 patients from the Children's Hospital of Philadelphia. SPADE was applied to a dataset with diagnoses coded using International Classification of Diseases (ICD) concepts and separately to a dataset with the ICD codes mapped to their corresponding expanded diagnostic clusters (EDCs). Common temporal condition patterns surrounding the initial diagnosis of pediatric asthma ascertained by SPADE from both the ICD and EDC datasets were compared. RESULTS: SPADE identified 36 unique diagnoses in the mapped EDC dataset, whereas only 19 were recognized in the ICD dataset. Temporal trends in condition diagnoses ascertained from the EDC data were not discoverable in the ICD dataset. DISCUSSION: Mining frequent temporal condition patterns from large electronic health record datasets may reveal previously unknown associations between diagnoses that could inform future research into causation or other relationships. Mapping sparsely coded medical concepts into homogenous groups was essential to discovering potentially useful information from our dataset. CONCLUSIONS: We expect that the presented methodology is applicable to the study of diagnostic trajectories for other clinical conditions and can be extended to study temporal patterns of other coded medical concepts such as medications and procedures.

OBJECTIVE: Patient transitions into home health care (HHC) often occur without the transfer of information needed for critical clinical decisions and the plan of care. Owing to a lack of universally implemented standards, there is wide variation in information transfer. We sought to characterize missing information at HHC admission. MATERIALS AND METHODS: We conducted a mixed methods study with 3 diverse HHC agencies. Focus groups with nurses at each agency identified what information supports patient care decisions at admission. Thirty-six in-home admissions with associated documentation review determined the available information. To inform information standards development for the HHC admission process, we compared the types of information desired and available to an international standard for transitions in care information, the Continuity of Care Document (CCD) enhanced with Office of the National Coordinator for Healthcare Information Technology summary terms (CCD/S). RESULTS: Three-quarters of the items from the focus groups mapped to the CCD/S. Regarding available information at admission, no observation included all CCD/S data items. While medication information was needed and often available for 4 important decisions, concepts related to patient medication self-management appeared in neither the CCD/S nor the admission documentation. DISCUSSION: The CCD/S mostly met HHC nurses' information needs and is recommended to begin to fill the current information gap. Electronic health record recommendations include use of a data standard: the CCD or the proposed, more parsimonious U.S. Core Data for Interoperability. CONCLUSIONS: Referral source and HHC agency adoption of data standards is recommended to support structured, consistent data and information sharing.

Connectibility affordances, or opportunities for a user to establish input/output cable connections, can be critical to the safety and usability of complex systems. To support model-based analyses, this research introduces a formal approach: Connectibility Affordance VErification, Modeling, and ENumeration (CAVEMEN). CAVEMEN is applicable to a human-environment system encompassing physical entities with specified properties, a user with specified motor abilities, and connectibility affordance instances involving different combinations of source-target connections. The modeling technique leverages object-oriented principles to define one instance of a connectibility affordance with respect to one unique combination of connection sources and targets. An inspection technique supports the enumeration of connectibility affordance instances that are desired (supporting a correct connection) and undesired (supporting an incorrect connection). A model checking technique aids in verifying accuracy, meaning the user can actualize desired affordance instances, and robustness, meaning undesired affordance instances never emerge. An XML-based grammar, a model checking syntax translation tool, and a linear temporal logic specification of accuracy and robustness support the analyses. We demonstrate CAVEMEN with a pacemaker system case study. The inspection aids in identifying nine desired and 18 undesired instances of chamber-port connectibility. The trace evaluation shows that accuracy and robustness depends on whether an entity property of interest can change in the environmental context. These results indicate that CAVEMEN shows promise for analyzing connectibility affordances of a safety-critical system.

User manual designers generally use written procedures, figures, and illustrations to convey procedural information to end users. However, ensuring that the instructions are accurate and unambiguous is difficult. With respect to accuracy, describing the task and under what conditions it should be conducted can be complicated by considerations such as the ordering of actions within a higher level task and the context under which tasks and lower level actions can be initiated, repeated, and completed. With respect to ambiguity, component and task level issues occur such as which portion of a component is relevant and what context constrains activity. To support accuracy and to decrease ambiguity, we propose a model-based approach coupled with model checking and visualization to aid in user manual development. Our approach integrates formal task-analytic and device models with safety specifications into a computational framework. We demonstrate the value of this approach using alarm troubleshooting instructions from the patient user manual of a left ventricular assist device. By encoding a formal task model, we revealed potential problems with task and device descriptions in the troubleshooting instructions. We also applied linear temporal logic and symbolic model checking to identify issues with the order of troubleshooting steps. Our framework provides insights into the use of formal methods for patient user manual evaluation.

Presents an editorial on the state of the journal.

Computational tools for modeling multi-agent work, including task work and teamwork, are needed to support the design of complex operational concepts involving human operators and automated systems. This paper presents a human performance model of task sharing with delegation where an agent can execute, delay, or delegate assigned task work according to its multi-tasking capacity and other attributes. Using a simulation framework for modeling multi-agent work, several case studies apply the task sharing with delegation model to the evaluation of an air transportation operational concept. The cases demonstrate how multi-tasking capacity, the delegation of task work, and the coordination of delegated task work impact the amount and time requirements of task work (on-time and delayed) and teamwork performed by agents. Thus, integrating human performance models in the simulation framework can inform operational concept development for multi-agent systems.

Quality assessment is the focus of many health care initiatives. Yet it is not well understood how the type of information used in decision support tools to enable judgments of quality based on data impacts the accuracy, consistency and reliability of judgments made by physicians. Comparative pooled information could allow physicians to judge the quality of their practice by making comparisons to other practices or other specific populations of patients. In this study, resident physicians were provided with varying types of information derived from pooled patient data sets: quality component measures at the individual and group level, a qualitative interpretation of the quality measures using percentile rank, and an aggregate composite quality score. 32 participants viewed thirty quality profiles consisting of information applicable to the practice of thirty de-identified resident physicians. Those provided with quality component measures and a qualitative interpretation of the quality measures (rankings) judged quality of care more similarly to experts and were more internally consistent compared to participants who were provided with quality component measures alone. Reliability between participants was significantly less for those who were provided with a composite quality score compared to those who were not.

Mapping eye tracking data for moving targets on computer displays is a difficult task due to the characteristics of the moving targets and the limited capabilities of the eye tracking systems. We introduce 1) a design concept to map eye fixations to moving targets, and 2) an algorithm that generates “target-based” eye tracking data that determines eye fixation order and duration specific to the moving targets. The mapping reduces the number of areas of interest (AOIs) required for the analysis by trading off AOI size and accuracy in correspondence with the targets of interest. Our resulting methodology was applied to a case study with an air traffic controller conducting a conflict detection task with multiple moving aircraft on a simulated radar screen. Benchmarking results show that a limited reduction in accuracy when considering the larger AOIs and the number of fixations on the targets. The methodology shows promise for supporting the analysis of eye tracking data for tracking multiple moving targets.

In systems that support dynamic allocation of work across human and autonomous agents, mechanisms for the delegation of authority are required. A case study inspired by the Überlingen mid-air collision is presented demonstrating an instantiation of a delegation assignment and coordination model. The scenario includes multiple delegation assignments issued from two sets of delegating agents to the same set of adopting agents. For one aircraft, two delegation assignments are made: a clearance from an air traffic controller and an opposing resolution from flightdeck traffic collision alerting automation. For the other aircraft, one delegation assignment is made (a resolution from flightdeck traffic collision alerting automation that is coordinated with the other traffic's automation and that makes the effect of the air traffic control clearance hazardous). The example demonstrates how the agents involved in Überlingen assigned and accepted multiple, coupled delegations and highlights issues arising from available information and the timing of assignments that may interfere with adherence to coordination procedures. This example indicates the need to extend methods for modeling of delegation assignment and coordination in order to inform the design of autonomous agents, associated procedures, and decision support.

States) were submitted to the THMS in 2013.Some authors appeared not to know about the changes to the scopes of the journals of the IEEE Systems, Man, and Cybernetics Society (SMCS) as 117 manuscripts submitted to THMS were rejected due to lack of fit.When considering the remaining manuscripts, 26 have been accepted and 101 have been rejected, yielding an acceptance rate for 2013 of 20.4% (including the 117 would bias the acceptance rate to an artificially low 6.8%).The rest are either under review or

Physician handoff of care is a mechanism for transferring patient information, responsibility, and authority from one set of caregivers to another. At shift change at a hospital, residents going off shift handoff patients to those coming on shift. There are limited handoff of care tools that facilitate the handover process by condensing patient information in reports that can be referenced during the handoff of care and used during patient care as cognitive artifacts. This effort works to address information, data entry and reporting requirements for a resident handoff of care tool that would support transfer of information as well as patient care.

Incomplete and/or unclear transfer of patient information can adversely affect the quality of patient care. This study investigated what/how information is exchanged between resident physicians at shift change. 15 sign-outs consisting of 209 patients were observed. Ten categories of patient information, four types of communication, and three sources of distraction were developed to characterize sign-out. The average sign-out lasted 34.4 minutes (standard deviation of 15.9) with 13.9 patients (standard deviation of 3.5) discussed. 23.2% of the time did not focus on patient care. When discussing patient care, 87.2% included one-way information transfer from the resident going off call. 32.2% of the information consisted of prior patient background. 12.1% was spent discussing what actions should/may occur overnight. Residents discussed the current physical condition for 35.4% of the patients, current medications for 62.7%, and contingency plans for 17.7% of patients. Potential improvements should focus on training residents on how to conduct handovers.

Computationally tractable methods are required to identify the potential contribution of sensors. This paper presents computationally tractable methods to measure radar coverage. Two measures are introduced. One is covered voxels which measures an upper bound of the number of distinct high quality values that can be returned from the radar system for a weather feature. The other is the high quality feature coverage area percentage that defines what percentage of the weather feature is sensed by covered voxels. Five methods (2 2D and 3 3D algorithms) are presented to help calculate the data required to determine the values for these measures. The algorithms are used on two systems to test drive the framework including run-time performance measures for the algorithms themselves. For both 2D and 3D cases, the approximations helped with respect to computation time. In addition, the feature approximation/3D volume coverage algorithm (using 3D-grid) greatly reduced the computation time.

OBJECTIVE: To characterize question types that residents received on overnight shifts and what information sources were used to answer them. MATERIALS AND METHODS: Across 30 overnight shifts, questions asked of on-call senior residents, question askers' roles, and residents' responses were documented. External sources were noted. RESULTS: 158 of 397 questions (39.8%) related to the plan of care, 53 (13.4%) to medical knowledge, 48 (12.1%) to taskwork knowledge, and 44 (11.1%) to the current condition of patients. For 351 (88.4%) questions residents provided specific, direct answers or visited the patient. For 16 of these, residents modeled or completed the task. For 216 questions, residents used previous knowledge or their own clinical judgment. Residents solicited external information sources for 118 questions and only a single source for 77 (65.3%) of them. For the 118, most questions concerned either the plan of care or the patient's current condition and were asked by interns and nurses (those with direct patient care responsibilities). DISCUSSION: Resident physicians serve as an information system and they often specifically answer the question using previous knowledge or their own clinical judgment, suggesting that askers are contacting an appropriately knowledgeable person. However, they do need to access patient information such as the plan of care. They also serve an educator role and answer many knowledge-related questions. CONCLUSIONS: As synchronous verbal communications continue to be important pathways for information flow, informaticians need to consider the relationship between such communications and workflow in the development of healthcare support tools.

The purpose of the framework introduced here is to support the development of evaluation scenarios that are capable of assessing system level performance while considering the system, the humans that interact with it, and the environment. The following five step framework is presented and applied to a pilot self separation task: 1) identify entities critical to system design, development, and operation and define their goals and properties as they relate to the system being studied; 2) define a subset of functionality for evaluation (define an execution sequence); 3) map entity properties to the execution sequence to identify independent variables; 4) translate entity goals into a set of system goals that can be used to identify dependent measures; and 5) iterate through each step to ensure the models produced are internally consistent.

The healthcare system is moving from one primary physician who assumes responsibility for each patient to a more team-based approach. Thus, assessing team communication is critical. This study characterizes and assesses the quality of hospitalist handover communications at shift change using the literature recommended content and language form elements. Quality handovers should contain the following content: patient identifiers, active issues, and care plans. Quality handovers also should include utterances in the following language forms: explanations, rationales, and directives. Interviews, observation, recording, and conversation analysis of hospitalist handover communications were used. Hospitalist handover utterances were assigned both content and language form codes. The proportion of quality element verbalization across all patient handovers was calculated. In addition, the impact of patient factors (new admission, new problem, acuity level) and handover receiver knowledge on the inclusion of quality elements was examined. The 106 individual patient handovers across 16 handover sessions were recorded. 39% contained all six quality elements. While the majority of handovers contained five out of six quality elements, only 48% included directives. There was also no difference in the inclusion of quality elements based on patient factors or handover receiver knowledge. Hospitalist handovers are lacking in directives. Efforts to improve handovers through enhanced electronic medical record systems and training may need to expand to hospitalists and other attending level physicians.

Human-automation interaction, including erroneous human behavior, is a factor in the failure of complex, safety-critical systems. This paper presents a method for automatically generating formal task analytic models encompassing both erroneous and normative human behavior from normative task models, where the misapplication of strategic knowledge is used to generate erroneous behavior. Resulting models can be automatically incorporated into larger formal system models so that safety properties can be formally verified with a model checker. This allows analysts to prove that a human-automation interactive system (as represented by the formal model) will or will not satisfy safety properties with both normative and generated erroneous human behavior. Benchmarks are reported that illustrate how this method scales. The method is then illustrated with a case study: the programming of a patient-controlled analgesia pump. In this example, a problem resulting from a generated erroneous human behavior is discovered. The method is further employed to evaluate the effectiveness of different solutions to the discovered problem. The results and future research directions are discussed.

Failures in complex systems controlled by human operators can be difficult to anticipate because of unexpected interactions between the elements that compose the system, including human-automation interaction (HAI). HAI analyses would benefit from techniques that support investigating the possible combinations of system conditions and HAIs that might result in failures. Formal verification is a powerful technique used to mathematically prove that an appropriately scaled model of a system does or does not exhibit desirable properties. This paper discusses how formal verification has been used to evaluate HAI. It has been used to evaluate human-automation interfaces for usability properties and to find potential mode confusion. It has also been used to evaluate system safety properties in light of formally modeled task analytic human behavior. While capable of providing insights into problems associated with HAI, formal verification does not scale as well as other techniques such as simulation. However, advances in formal verification continue to address this problem, and approaches that allow it to complement more traditional analysis methods can potentially avoid this limitation.

Due to the NP-hardness of factor-graph optimization, obtaining exact solutions to problems with a large variable domain is generally not possible. Max-sum (max-product) belief propagation (BP) is a distributed message-passing heuristic that has found popularity due to its ability to generate approximate solutions to such factor-graph problems in a distributed fashion. Because max-sum BP generally provides no indication of solution quality, researchers have sought alternative algorithms to generate approximate (and, in some cases, exact) solutions, the most successful of which operate on a relaxation of the integer programming form of an equivalent maximum a posteriori estimation problem. While such linear-programming-based algorithms perform well in empirical studies, there are limits to the variable domain size for which they are tractable. Via a case study in weather-radar coordination, we demonstrate that the decentralized max-sum BP algorithm remains useful for generating quality solutions to problems with a large variable domain. Our custom simulation tool facilitates a comparison of the performance of algorithms with respect to adaptive weather-radar scanning resource allocation across three weather scenarios. In addition to no adaptive scanning, the algorithms include four max-sum-BP-based algorithms: decentralized distributed max-sum BP, self-terminating tree-based bounded approximation, tabu search implemented in a centralized fashion, and a combination of the latter two. Performance is measured by the end-user utility for all algorithms and by two types of approximation ratios for the tree-based bounded approximation. BP-based decentralized algorithms are found to exhibit comparable performance with a centralized algorithm and superior performance to no adaptive scanning. Furthermore, our analysis demonstrates that max-sum BP is capable of generating solutions within 67% of optimal (and typically much better) across the weather scenarios.

Breakdowns in complex systems often occur as a result of system elements interacting in ways unanticipated by analysts or designers. The use of task behavior as part of a larger, formal system model is potentially useful for analyzing such problems because it allows the ramifications of different human behaviors to be verified in relation to other aspects of the system. A component of task behavior largely overlooked to date is the role of human-human interaction, particularly human-human communication in complex human-computer systems. We are developing a multi-method approach based on extending the Enhanced Operator Function Model language to address human agent communications (EOFMC). This approach includes analyses via theorem proving and future support for model checking linked through the EOFMC top level XML description. Herein, we consider an aviation scenario in which an air traffic controller needs a flight crew to change the heading for spacing. Although this example, at first glance, seems to be one simple task, on closer inspection we find that it involves local human-human communication, remote human-human communication, multi-party communications, communication protocols, and human-automation interaction. We show how all these varied communications can be handled within the context of EOFMC.

Researchers have conducted few studies of the joint decision-making processes of forecasters and emergency managers during severe weather events. Emergency managers are difficult to study due to lack of standardization and the jurisdictional nature of their work This research describes an operational concept, requirements, design, and implementation of a forecaster, communicator and emergency computer system and work flow that support the study of forecaster-emergency interactions. The experiment supported by the system and workflow assume that forecasters are available to be experiment participants (foregoing their typical jobs responsibilities) while emergency managers must continue to support their job responsibilities while simultaneously considering the experimental protocol. Thus the operational concept imposes limited additional efforts on the part of the emergency managers. Toward that end, we take advantage of communication methods such as Twitter as well as those already used in the field (such as chat). Limitations of this current effort and ideas for future work are also presented.

Human-automation interaction, including erroneous human behavior, is a factor in the failure of complex, safety-critical systems. This paper presents a method for automatically generating task analytic models encompassing both erroneous and normative human behavior from normative task models by manipulating modeled strategic knowledge. Resulting models can be automatically translated into larger formal system models so that safety properties can be formally verified with a model checker. This allows analysts to prove that a human automation-interactive system (as represented by the formal model) will or will not satisfy safety properties with both normative and generated erroneous human behavior. This method is illustrated with a case study: the programming of a patient-controlled analgesia pump. In this example, a problem resulting from a generated erroneous human behavior is discovered and a potential solutions is explored. Future research directions are discussed.

A framework is presented for comparing sensor coverage between networks of different spatial and temporal resolutions (long-range vs. short-range adaptive), in the context of their operational constraints. A simulation case study evaluates five levels of spacing between adaptive sensors (30km, 35km, 40km, 45km, 50km) in the context of neighboring long-range sensors that provide complementary data. Total system performance is assessed with measures of sensor coverage quality based on the number of radar nodes, tilts and cycles and the fraction of the analysis region's horizontal and vertical domain scanned. Using a squall line weather scenario, results show that 3D scanning performance is significantly lower with 45km and 50km spaced adaptive sensors than with 30km spaced nodes. Comparison with complementary long-range sensors shows that short-range adaptive sensors perform significantly greater at altitudes below 20000 ft. Network performance at all altitudes is shown to be significantly improved by considering both network types together when compared to either network alone. The analysis framework presented can help inform decision makers of the expected value added by deploying sensors relative to their type and quantity, for the context in which they will be operating.

Formal methods are typically used in the analysis of complex system components that can be described as “automated” (digital circuits, devices, protocols, and software). Human-automation interaction has been linked to system failure, where problems stem from human operators interacting with an automated system via its controls and information displays. As part of the process of designing and analyzing human-automation interaction, human factors engineers use task analytic models to capture the descriptive and normative human operator behavior. In order to support the integration of task analyses into the formal verification of larger system models, we have developed the enhanced operator function model (EOFM) as an Extensible Markup Language-based, platform- and analysis-independent language for describing task analytic models. We present the formal syntax and semantics of the EOFM and an automated process for translating an instantiated EOFM into the model checking language Symbolic Analysis Laboratory. We present an evaluation of the scalability of the translation algorithm. We then present an automobile cruise control example to illustrate how an instantiated EOFM can be integrated into a larger system model that includes environmental features and the human operator's mission. The system model is verified using model checking in order to analyze a potentially hazardous situation related to the human-automation interaction.

Model checking is a type of automated formal verification that searches a system model's entire state space in order to mathematically prove that the system does or does not meet desired properties. An output of most model checkers is a counterexample: an execution trace illustrating exactly how a specification was violated. In most analysis environments, this output is a list of the model variables and their values at each step in the execution trace. We have developed a language for modeling human task behavior and an automated method which translates instantiated models into a formal system model implemented in the language of the Symbolic Analysis Laboratory (SAL). This allows us to use model checking formal verification to evaluate human-automation interaction. In this paper we present an operational concept and design showing how our task modeling visual notation and system modeling architecture can be exploited to visualize counterexamples produced by SAL. We illustrate the use of our design with a model related to the operation of an automobile with a simple cruise control.

How to design and to evaluate adaptive distributed sensor networks with respect to end user decision making is an important topic in human-automation interaction. This study evaluates four resource allocation strategies (static automation, adaptive sensing, mixed-initiative control, and control incorporating end user task analytic information) in a simulated prototype weather sensor network. Performance of each sensor network resource allocation strategy was assessed with measures of sensor coverage quality over a target geographic region of interest. Coverage measures calculated from the number of radar nodes, tilts and cycles were derived for partial (radar scan touches at least part of the analysis region) and complete coverage of the region by single and multiple (two or more) radars. Single and multi-radar coverage area scores were also derived from the cumulative fraction of the analysis regions area that was scanned. Using three weather scenarios, results show that static automation significantly outperforms the other strategies in all cases using measures of single and multi-radar complete coverage and for a uniformly distributed weather event using single and multi-radar coverage scores. Otherwise the three strategies outperform static automation. As the performance of resource allocation strategies are sensitive to the weather events themselves as well as the measures used to evaluate them, the design of next generation of algorithms should consider these performance distinctions.

Electronic medical record and separate sign-out system data (code status, location, medications and laboratory results) from 60 patients were compared to assess the impact of parallel data systems. In addition, a survey was administered to 74 Internal Medicine residents to characterize what elements they self-report using in their sign-out system. The survey results indicated that residents use the separate sign-out system to support hand-off of care, but also patient care. The data comparison showed that while code status data matched for all patients, 28.3% of patients had a location entered into the sign-out system that did not match the medical record. 46 (76.7%) patients had a medication reported in the sign-out report that did not match the medical record. Of 180 laboratory values compared, 130 (72%) matched the most recent value or a value within the past 24 hours. Using a criterion of “medically relevant matches”, a test of proportion indicated that significantly more laboratory values agree, thus supporting the hypothesis that physicians are less likely to update data when changes are not medically relevant. To reduce workload and the amount of inaccurate information being passed to physicians, sign-out functionality should be integrated into electronic medical record systems.

BACKGROUND: While the pairwise alignments produced by sequence similarity searches are a powerful tool for identifying homologous proteins - proteins that share a common ancestor and a similar structure; pairwise sequence alignments often fail to represent accurately the structural alignments inferred from three-dimensional coordinates. Since sequence alignment algorithms produce optimal alignments, the best structural alignments must reflect suboptimal sequence alignment scores. Thus, we have examined a range of suboptimal sequence alignments and a range of scoring parameters to understand better which sequence alignments are likely to be more structurally accurate. RESULTS: We compared near-optimal protein sequence alignments produced by the Zuker algorithm and a set of probabilistic alignments produced by the probA program with structural alignments produced by four different structure alignment algorithms. There is significant overlap between the solution spaces of structural alignments and both the near-optimal sequence alignments produced by commonly used scoring parameters for sequences that share significant sequence similarity (E-values < 10-5) and the ensemble of probA alignments. We constructed a logistic regression model incorporating three input variables derived from sets of near-optimal alignments: robustness, edge frequency, and maximum bits-per-position. A ROC analysis shows that this model more accurately classifies amino acid pairs (edges in the alignment path graph) according to the likelihood of appearance in structural alignments than the robustness score alone. We investigated various trimming protocols for removing incorrect edges from the optimal sequence alignment; the most effective protocol is to remove matches from the semi-global optimal alignment that are outside the boundaries of the local alignment, although trimming according to the model-generated probabilities achieves a similar level of improvement. The model can also be used to generate novel alignments by using the probabilities in lieu of a scoring matrix. These alignments are typically better than the optimal sequence alignment, and include novel correct structural edges. We find that the probA alignments sample a larger variety of alignments than the Zuker set, which more frequently results in alignments that are closer to the structural alignments, but that using the probA alignments as input to the regression model does not increase performance. CONCLUSIONS: The pool of suboptimal pairwise protein sequence alignments substantially overlaps structure-based alignments for pairs with statistically significant similarity, and a regression model based on information contained in this alignment pool improves the accuracy of pairwise alignments with respect to structure-based alignments.

Patient sign-out is a mechanism for transferring information, responsibility, and/or authority from one set of caregivers to another. Little research has addressed what information should be communicated during sign-out and how sign-out should be conducted and evaluated. As hospital residents conduct many sign-outs and have limited time in general, targeted web-based training and evaluation have the potential to enhance Graduate Medical Education. However there are no web-based training systems for this very important skill. This paper presents the operational concept and system requirements for a web-based sign-out training system. It discusses an initial functional prototype. Results of a heuristic evaluation and an assessment of areas for improvement are presented.

Task analytic models are extremely useful for human factors and systems engineers. Unfortunately, there is no standard language for describing task models. We present an XML-based task analytic modeling language. The language incorporates features from operator function model and extends them with additional task sequencing options and conditional constraints. This language's use is illustrated via a radio alarm clock example. In addition, parsing, visualization, and development tools are discussed.

This study examines the impact of increased spatio-temporal resolution weather radar data on the judgment accuracy and warning decisions of forecasters. In a static part-task setting, weather forecasters were provided with high resolution radar data in addition to conventional radar data and asked to forecast ground level winds two to five minutes into the future. When given these additional data, subjects significantly increased wind speed assessments, decreased absolute error, increased confidence ratings, and changed the number of affirmative warning decisions.

The Engineering Research Center for Collaborative Adaptive Sensing of the Atmosphere (CASA) is creating a new paradigm for weather observation based on low cost, densely spaced networks of X-band radars. These networks adapt their scanning strategy based on the evolving weather and user needs for data. This paper presents the results of an evaluation by National Weather Service forecasters and academic researchers of the dynamically reconfigurable radar scanning patterns that operated in CASA's prototype test bed in southwest Oklahoma in spring 2007. The evaluation demonstrates that a pilot group of users were satisfied overall with CASA's scanning patterns. The evaluation also uncovered needed improvements to the scanning strategy that have been subsequently implemented. Through this iterative cycle of design, implementation, evaluation, we have demonstrated the flexibility of the system architecture and our ability to modify existing and add new capabilities to increase the benefits of CASA radar systems.

Human-automated judge learning (HAJL) is a methodology providing a three-phase process, quantitative measures, and analytical methods to support design of information analysis automation. HAJL's measures capture the human and automation's judgment processes, relevant features of the environment, and the relationships between each. Specific measures include achievement of the human and the automation, conflict between them, compromise and adaptation by the human toward the automation, and the human's ability to predict the automation. HAJL's utility is demonstrated herein using a simplified air traffic conflict prediction task. HAJL was able to capture patterns of behavior within and across the three phases with measures of individual judgments and human-automation interaction. Its measures were also used for statistical tests of aggregate effects across human judges. Two between-subject manipulations were crossed to investigate HAJL's sensitivity to interventions in the human's training (sensor noise during training) and in display design (information from the automation about its judgment strategy). HAJL identified that the design intervention impacted conflict and compromise with the automation, participants learned from the automation over time, and those with higher individual judgment achievement were also better able to predict the automation.

Spatial awareness is important in domains where safety hinges on human operators keeping track of the relative locations of objects in their environments. While a variety of subjective and judgment-based measures have been used to evaluate spatial awareness, none have probed all three of its levels: 1) identification of environmental objects, 2) their current locations relative to the operator, and 3) their relative positions over time. This work compares new judgment based measures of spatial awareness that probe all three levels of spatial awareness to conventional subjective measures. In the evaluation of 14 configurations of Synthetic Vision Systems head down displays (7 terrain textures and 2 Fields of View (FOVs)), 18 pilots made 4 types of judgments (relative angle, distance, height, and a beam time) regarding the location of terrain points displayed in 112 5-second, non-interactive simulations. They also provided subjective awareness and SA-SWORD measures. ANOVA analyses revealed that comparable results were found between display configurations that produced the minimum error in judgments and those that received the highest subjective ratings. However, none of the subjective measures were correlated with judgment error. Thus, given that the judgment based measures were explicitly designed to measure all three levels of spatial awareness, the subjective measures may not be measuring spatial awareness.

Weather-related judgment and decision-making are critical in many domains. Weather assessments typically involve integrating information from multiple sources with varying content, format, and update rate. One way to study weather-related decision-making is to have participants engage in job related tasks using simulated real-time weather scenarios. This paper discusses a part-task scenario support tool that has been developed to facilitate the process of presenting weather information in simulated real-time. The paper also presents an illustrative example where the tool supported a study of weather decision-making by emergency managers.

The evaluation of human-centered systems can be performed using a variety of different methodologies. This paper describes a human-centered systems' evaluation methodology where participants watch 5-second non-interactive videos of a system in operation before supplying judgments and subjective measures based on the information conveyed in the videos. This methodology was used to evaluate the ability of different textures and fields of view to convey spatial awareness in synthetic vision systems (SVS) displays. It produced significant results for both judgment based and subjective measures. This method is compared to other methods commonly used to evaluate SVS displays based on cost, the amount of experimental time required, experimental flexibility, and the type of data provided.

Development of web-based training tools requires the collaboration of instructional system designers, domain subject matter experts, web developers, and software engineers. This paper discusses the enhancements to a web-based training architecture to enable collaboration among these contributors. The enhancements allow domain subject matter experts to view the training material and to provide feedback. The paper provides an illustrative example where the system was used during the development of a web-based training tool for flight dispatchers.

Physician sign-out is a mechanism for transferring patient information, responsibility, and/or authority between care givers at a shift change. Despite being an important process, little research has examined the content and process of conducting sign-out. This paper describes the development and application of a support tool used to study and analyze physician sign-out. The tool supports the identification and analysis of 10 types of patient information, five types of physician communication, and three sources of distraction to physicians. The tool was applied during observations of sign-out on acute care pediatric wards to form a baseline of the current sign-out process and the information exchanged between physicians to identify areas in need of procedural support and training interventions.

There is a lack of educational technology to support cognitive systems engineering topics such as models of human performance in dynamic environments. This paper describes the Cognitive Systems Engineering Educational Software (CSEES) system, an integrated toolset designed to facilitate curricula related to human judgment and decision-making performance modeling and evaluation. CSEES provides students with the means to generate and analyze performance data using multiple methods. It also includes documentation and tutorials. Its flexible design facilitates adding new judgment and decision making task environments. The paper describes the initial system implementation as a Microsoft Excel add-on, a preliminary evaluation as part of a graduate engineering course, and planned future work.

Being able to incrementally define and test knowledge bases for intelligent systems is desirable. However, as more knowledge is added, the knowledge engineer must ensure that unwanted interactions between the existing and additional knowledge do not occur. One knowledge-based monitoring system, Hazard Monitor (HM), provides the ability to add knowledge incrementally. HM's architecture includes tailorable components that allow the knowledge engineer to eliminate unwanted knowledge interactions. HM also includes knowledge-base development tools to facilitate initial and incremental knowledge-base development. This paper describes HM's architecture and knowledge structures plus its knowledge-base development tools that facilitate the knowledge engineering process. To illustrate the benefits of the architecture, we present aviation examples based on our implemented prototypes.

An experiment using a simulated air traffic conflict prediction task with sensor noise was conducted with one between subjects manipulation and the sensor noise level. Measures from signal detection theory, fuzzy signal detection theory, and judgment analysis were calculated and analyzed. Gender, noise, and session (experience) effects and their two-way interactions were investigated. As different effects were significant for different measures, analysts should take care in relying on the results from any single measure when analyzing human judgment performance.

Military and general aviation training organizations continually have to employ less experienced instructors. As this trend is expected to continue, there is a need to provide ways to help less experienced instructors do their jobs more effectively and efficiently. The paper presents the architecture for an intelligent instructor pilot decision support system that helps instructors during simulated missions instruction, post-flight debriefing, and performance evaluation.