We discuss the KDD process in "data-flow" environments, where unstructured and time dependent data can be processed into various levels of structured and semanticallyrich forms for analysis tasks.Using network intrusion detection as a concrete application example, we describe how to construct models that are both acczLrate in describing the underlying concepts, and efficient when used to analyze data in real-time.We present procedures for analyzing frequent patterns from lower level data and constructing appropriate features to formulate higher level data.The features generated from various levels of data have different computational costs (in time and space).We show that in order to minimize the time required in using the classification models in a real-time environment, we can exploit the "necessary conditions" associated with the lowcost features to determine whether some high-cost features need to be computed and the corresponding classification rules need to be checked.We have applied our tools to the problem of building network intrusion detection models.We report our experiments using the network data provided as part of the 1998 DARPA Intrusion Detection Evaluation program.We also discuss our experience in using the mined models in NFR, a real-time network intrusion detection system.

There is often the need to update an installed intrusion detection system (IDS) due to new attack methods or upgraded computing environments. Since many current IDSs are constructed by manual encoding of expert knowledge, changes to IDSs are expensive and slow. We describe a data mining framework for adaptively building Intrusion Detection (ID) models. The central idea is to utilize auditing programs to extract an extensive set of features that describe each network connection or host session, and apply data mining programs to learn rules that accurately capture the behavior of intrusions and normal activities. These rules can then be used for misuse detection and anomaly detection. New detection models are incorporated into an existing IDS through a meta-learning (or co-operative learning) process, which produces a meta detection model that combines evidence from multiple models. We discuss the strengths of our data mining programs, namely, classification, meta-learning, association rules, and frequent episodes. We report on the results of applying these programs to the extensively gathered network audit data for the 1998 DARPA Intrusion Detection Evaluation Program.