This paper overviews an end-to-end learning-based approach to the rapid development of practical cognitive assistants for intelligence analysis. A learning agent shell has been trained by a knowledge engineer with general evidence-based reasoning knowledge for intelligence analysis. This agent is further trained by an expert analyst how to analyze complex hypotheses from a given intelligence analysis domain. The resulting cognitive assistant is used by a typical analyst to rapidly analyze hypotheses from agent's area of expertise. During its use, the agent continues to learn reasoning patterns from its user. This approach has been implemented and practical agents have been developed and used. This is a significant application of machine learning to agents development in intelligence analysis that can be generalized to many other domains involving evidence-based reasoning, including medicine, law, and science.

Our research addresses the issue of developing knowledge-based agents that capture and use the problem solving knowledge of subject matter experts from diverse application domains. This paper emphasizes the use of negative examples in agent learning by presenting several strategies for capturing expert's knowledge when the agent fails to correctly solve a problem. These strategies have been implemented into the disciple learning agent shell and used in complex application domains such as intelligence analysis, center of gravity determination, and emergency response planning.

This paper presents recent results on developing learning agents that can be taught by subject matter experts how to solve problems, through examples and explanations. It introduces the lazy rule refinement method where the expert modifies an example generated by a learned rule. In this case the agent has to decide whether to modify the rule (if the modification applies to all the previous positive examples) or to learn a new rule. However, checking the previous examples would be disruptive or even impossible. The lazy rule refinement method provides an elegant solution to this problem, in which the agent delays the decision whether to modify the rule or to learn a new rule until it accumulated enough examples during the follow-on problem solving process. This method has been incorporated into the disciple learning agent shell and used in the complex application areas of center of gravity analysis and intelligence analysis

Over the years we have developed the Disciple theory, methodology, and family of tools for building knowledge-based agents. This approach consists of developing an agent shell that can be taught directly by a subject matter expert in a way that resembles how the expert would teach a human apprentice when solving problems in cooperation. This paper presents the most recent version of the Disciple approach and its implementation in the Disciple–RKF (rapid knowledge formation) system. Disciple–RKF is based on mixed-initiative problem solving, where the expert solves the more creative parts of the problem and the agent solves the more routine ones, integrated teaching and learning, where the agent helps the expert to teach it, by asking relevant questions, and the expert helps the agent to learn, by providing examples, hints, and explanations, and multistrategy learning, where the agent integrates multiple learning strategies, such as learning from examples, learning from explanations, and learning by analogy, to learn from the expert how to solve problems. Disciple–RKF has been applied to build learning and reasoning agents for military center of gravity analysis, which are used in several courses at the US Army War College.

This paper presents the Disciple-RKF methodology for rapid development of large knowledge bases which relies on importing ontological knowledge from existing knowledge repositories, on parallel development of separate knowledge bases by subject matter experts, and on the merging of these knowledge bases into a high performance integrated knowledge base. The paper discusses several issues related to ontology import and merging, and presents the results of a successful knowledge base development and integration experiment performed at the US Army War College.

This paper presents current results in developing a practical approach, methodology and tool, for the development of knowledge bases and agents by subject matter experts, with limited assistance from knowledge engineers. This approach is based on mixed-initiative reasoning that integrates the complementary knowledge and reasoning styles of a subject matter expert and a learning agent, and on a division of responsibilities for those elements of knowledge engineering for which they have the most aptitude. The approach was evaluated at the US Army War College, demonstrating very good results and a high potential for overcoming the knowledge acquisition bottleneck.

This paper describes Disciple, an Artificial Intelligence based decision aid which subject-matter experts can train and use when making decisions under stressful, complex, and constrained conditions. The tool was developed and used under the Defense Advanced Research Projects Agency's High Performance Knowledge Base and Rapid Knowledge Formation programs. Some domains in which the tool would be applicable are described, with particular emphasis on military battle planning. The paper concludes with a discussion of future trends in decision-support application tools.

The paper presents an interactive multistrategy learning system (MTLS) that extends and refines knowledge bases by learning from input examples, discovering new knowledge, and cooperating with a user. The use of the multistrategy learning approach based on plausible justification trees allows MTLS to perform learning tasks that are beyond the capability of a single strategy learning method. A goal driven knowledge discovery method has been developed and integrated into MTLS to produce additional knowledge needed by the system. MTLS also allows a human expert to guide it to refine and extend the knowledge base. This cooperation between a human expert and the learner enables the system to perform tasks that are intrinsically difficult for an autonomous system. The resulting knowledge base may include new rules discovered from data, as well as revised rules, and new facts learned by analogy. MTLS has been developed as a tool to be used by a domain expert to build a knowledge base to reduce the need for assistance from a knowledge engineer.

The ability to build intelligent agents is significantly constrained by the knowledge acquisition effort required. Many iterations by human experts and knowledge engineers are currently necessary to develop knowledge‐based agents with acceptable performance. We have developed a novel approach, called Disciple, for building intelligent agents that relies on an interactive tutoring paradigm, rather than the traditional knowledge engineering paradigm. In the Disciple approach, an expert teaches an agent through five basic types of interactions. Such rich interaction is rare among machine learning (ML) systems, but is necessary to develop more powerful systems. These interactions, from the point of view of the expert, include specifying knowledge to the agent, giving the agent a concrete problem and its solution that the agent is to learn a general rule for, validating analogical problems and solutions proposed by the agent, explaining to the agent reasons for the validation, and being guided to provide new knowledge during interaction. In this article, we illustrate these basic learning interactions between an expert and an intelligent agent in the context of teaching the agent for military training simulations.

A method for the automation of knowledge acquisition that is viewed as a process of incremental extension, updating, and improvement of an incomplete and possibly partially incorrect knowledge base of an expert system is presented. The knowledge base is an approximate representation of objects and inference processes in the expertise domain. Its gradual development is guided by the general goal of improving this representation to consistently integrate new input information received from the human expert. The knowledge acquisition method is presented as part of a methodology for the automation of the entire process of building expert systems, and is implemented in the system NeoDISCIPLE. The method promotes several general ideas for the automation of knowledge acquisition, such as understanding-based knowledge extension, knowledge acquisition through multistrategy learning, consistency-driven concept formation and refinement, closed-loop learning, and synergistic cooperation between a human expert and a learning system.>

An approach to concept learning from examples and concept learning by observation is presented that is based on a intuitive notion of conceptual distance between examples (concepts) and combines symbolical and numerical methods. The approach is based on the observation that very different examples generalize to an expression that is very far from each of them, while identical examples generalize to themselves. Following this idea the authors propose some domain-independent and intuitively justified estimates for the conceptual distance. A hierarchical conceptual clustering algorithm that groups objects so as to maximize the cohesiveness (a reciprocal of the conceptual distance) of the clusters is presented. It is shown that conceptual clustering can improve learning from complex examples describing objects and the relation between them.>