BACKGROUND: Evidence-based clinical guidelines (GLs) are essential for standardizing care, yet often difficult to apply. Most clinical decision support systems (CDSSs) assume continuous application, which misaligns with the episodic nature of real-world workflows. OBJECTIVES: To design, implement, and evaluate e-Picard, a CDSS that provides GL-based recommendations through episodic, intermittent, on-demand consultations. The system supports retrospective assessment of past care and prospective identification of required actions. The evaluation focused on system validity and on its potential, in a retrospective simulation on real-world data, to enhance staff adherence to the GLs and to assess the potential effect of varying the frequency of the consultations. METHODS: The system development involved three preprocessing steps: (1) acquisition of free-text GLs with domain experts; (2) modeling procedural logic as workflows; and (3) flattening these into declarative temporal patterns for retrospective quality assessment and prospective recommendations. At runtime, e-Picard analyzes offline patient data to identify missed actions, computes compliance using fuzzy logic, and generates context-specific recommendations. e-Picard was applied to pressure-ulcer (PU) and diabetes management (DM) GLs, adapted for episodic use. Technical validation was performed on records from 43 PU and 82 DM patients. A retrospective simulation using 1,000 patients per domain estimated potential increases in adherence under varying consultation frequencies. RESULTS: Technical manual validation showed high correctness (≥99 %) and completeness (up to 98 %), based on 3,110 PU and 12,538 DM data instances (i.e., clinical measurements or actions), across various clinical scenarios over two-week observation periods. Retrospective simulation covered 57,860 PU and 100,940 DM data instances with estimated adherence potentially increasing from 68 %-69 % to 89 %-97 % for PU and from 14 %-15 % to 60 %-87 % for DM, in the real-world data retrospective simulation, assuming full adherence of the staff to the system's recommendations, depending on the scenario. Higher consultation frequency yielded greater gains, and adherence variability across hospital units and patient subgroups was reduced. CONCLUSIONS: Episodic CDSSs can deliver accurate, context-aware recommendations in environments with intermittent use and incomplete data, with the potential, assuming that the real-world data retrospective simulation results hold, to enhance adherence and consistency in care.

Sepsis is a condition caused by the body's overwhelming and life-threatening response to infection, which can lead to tissue damage, organ failure, and finally death. Today, sepsis is one of the leading causes of mortality among populations in intensive care units (ICUs). Sepsis is difficult to predict, diagnose, and treat, as it involves analyzing different sets of multivariate time-series, usually with problems of missing data, different sampling frequencies, and random noise. Here, we propose a new dynamic-behavior-based model, which we call a Temporal Probabilistic proFile (TPF), for classification and prediction tasks of multivariate time series. In the TPF method, the raw, time-stamped data are first abstracted into a series of higher-level, meaningful concepts, which hold over intervals characterizing time periods. We then discover frequently repeating temporal patterns within the data. Using the discovered patterns, we create a probabilistic distribution of the temporal patterns of the overall entity population, of each target class in it, and of each entity. We then exploit TPFs as meta-features to classify the time series of new entities, or to predict their outcome, by measuring their TPF distance, either to the aggregated TPF of each class, or to the individual TPFs of each of the entities, using negative cross entropy. Our experimental results on a large benchmark clinical data set show that TPFs improve sepsis prediction capabilities, and perform better than other machine learning approaches.

OBJECTIVE: To analyze the longitudinal data of multiple patients and to discover new temporal knowledge, we designed and developed the Visual Temporal Analysis Laboratory (ViTA-Lab). In this study, we demonstrate several of the capabilities of the ViTA-Lab framework through the exploration of renal-damage risk factors in patients with diabetes type II. MATERIALS AND METHODS: The ViTA-Lab framework combines data-driven temporal data mining techniques, with interactive, query-driven, visual analytical capabilities, to support, in an integrated fashion, an iterative investigation of time-oriented clinical data and of patterns discovered in them. Patterns discovered through the data mining mode can be explored visually, and vice versa. Both analysis modes are supported by a rich underlying ontology of clinical concepts, their relations, and their temporal properties. The knowledge enables us to apply a temporal-abstraction pre-processing phase that abstracts in a context-sensitive manner raw time-stamped data into interval-based clinically meaningful interpretations, increasing the results' significance. We demonstrate our approach through the exploration of risk factors associated with future renal damage (micro-albuminuria and macro-albuminuria) and their relationship to the hemoglobin A1C (HbA1C ) and creatinine level concepts, in the longitudinal records of 22 000 patients with diabetes type II followed for up to 5 years. RESULTS: The iterative ViTA-Lab analysis process was highly feasible. Higher ranges of either normal albuminuria or normal creatinine values and their combination were shown to be significantly associated with future micro-albuminuria and macro-albuminuria. The risk increased given high HbA1C levels for women in the lower range of normal albuminuria, and for men in the higher range of albuminuria. CONCLUSIONS: The ViTA-Lab framework can potentially serve as a virtual laboratory for investigations of large masses of longitudinal clinical databases, for discovery of new knowledge through interactive exploration, clustering, classification, and prediction.

A popular use of crowdsourcing is to collect and aggregate individual worker responses to problems to reach a correct answer. This paper studies the relationship between the computation complexity class of problems, and the ability of a group to agree on a correct solution. We hypothesized that for NP-Complete (NPC) problems, groups would be able to reach a majority-based correct solution once it was suggested by a group member and presented to the other members, due to the "easy to verify" (i.e., verification in polynomial time) characteristic of this complexity class. In contrast, when posed with PSPACE-Complete (PSC) "hard to verify" problems (i.e., verification in exponential time), groups will not necessarily be able to choose a correct solution even if such a solution has been presented. Consequently, increasing the size of the group is expected to facilitate the ability of the group to converge on a correct solution when solving NPC problems, but not when solving PSC problems. To test this hypothesis we conducted preliminary experiments in which we evaluated people's ability to solve an analytical problem and their ability to recognize a correct solution. In our experiments, participants were significantly more likely to recognize correct and incorrect solutions for NPC problems than for PSC problems, even for problems of similar difficulties (as measured by the percentage of participants who solved the problem). This is a first step towards formalizing a relationship between the computationally complexity of a problem and the crowd's ability to converge to a correct solution to the problem.

Detecting unknown worms is a challenging task. Extant solutions, such as anti-virus tools, rely mainly on prior explicit knowledge of specific worm signatures. As a result, after the appearance of a new worm on the Web there is a significant delay until an update carrying the worm's signature is distributed to anti-virus tools. During this time interval a new worm can infect many computers and cause significant damage. We propose an innovative technique for detecting the presence of an unknown worm, not necessarily by recognizing specific instances of the worm, but rather based on the computer measurements. We designed an experiment to test the new technique employing several computer configurations and background applications activity. During the experiments 323 computer features were monitored. Four feature selection techniques were used to reduce the amount of features and four classification algorithms were applied on the resulting feature subsets. Our results indicate that using this approach resulted in exceeding 90% mean accuracy, and for specific unknown worms accuracy reached above 99%, using just 20 features while maintaining a low level of false positive rate.

Detecting unknown worms is a challenging task. Extant solutions, such as anti-virus tools, rely mainly on prior explicit knowledge of specific worm signatures. As a result, after the appearance of a new worm on the Web there is a significant delay until an update carrying the worm's signature is distributed to anti-virus tools. During this time interval a new worm can infect many computers and create significant damage. We propose an innovative technique for detecting the presence of an unknown worm, not necessarily by recognizing specific instances of the worm, but rather based on the computer measurements. We designed an experiment to test the new technique employing several computer configurations and background applications activity. During the experiments 323 computer features were monitored. Four feature selection techniques were used to reduce the amount of features and four classification algorithms were applied on the resulting feature subsets. Our results indicate that using this approach resulted, in above 90% average accuracy, and for specific unknown worms accuracy reached above 99%, using just 20 features while maintaining a low level of false positive rate

Detecting unknown malicious code (malcode) is a challenging task. Current common solutions, such as anti-virus tools, rely heavily on prior explicit knowledge of specific instances of malcode binary code signatures. During the time between its appearance and an update being sent to anti-virus tools, a new worm can infect many computers and cause significant damage. We present a new host-based intrusion detection approach, based on analyzing the behavior of the computer to detect the presence of unknown malicious code. The new approach consists on classification algorithms that learn from previous known malcode samples which enable the detection of an unknown malcode. We performed several experiments to evaluate our approach, focusing on computer worms being activated on several computer configurations while running several programs in order to simulate background activity. We collected 323 features in order to measure the computer behavior. Four classification algorithms were applied on several feature subsets. The average detection accuracy that we achieved was above 90% and for specific unknown worms even above 99%.

In this paper we describe the design and evaluation of the Temporal Knowledge Master, a graphical knowledge-acquisition (KA) tool used for entering the knowledge re-quired by any implementation of the Knowledge-Based Temporal Abstraction (KBTA) method. The KBTA method provides mechanisms that perform derivation of context-specific, interval-based abstract interpretations (also known as Temporal Abstractions) from raw time-stamped data, by using a domain-specific knowledge-base. The study evalu-ated the functionality and usability of the KA tool in the computer-network security domain.

OBJECTIVES: Study comparatively (1) concept-based search, using documents pre-indexed by a conceptual hierarchy; (2) context-sensitive search, using structured, labeled documents; and (3) traditional full-text search. Hypotheses were: (1) more contexts lead to better retrieval accuracy; and (2) adding concept-based search to the other searches would improve upon their baseline performances. DESIGN: Use our Vaidurya architecture, for search and retrieval evaluation, of structured documents classified by a conceptual hierarchy, on a clinical guidelines test collection. MEASUREMENTS: Precision computed at different levels of recall to assess the contribution of the retrieval methods. Comparisons of precisions done with recall set at 0.5, using t-tests. RESULTS: Performance increased monotonically with the number of query context elements. Adding context-sensitive elements, mean improvement was 11.1% at recall 0.5. With three contexts, mean query precision was 42% +/- 17% (95% confidence interval [CI], 31% to 53%); with two contexts, 32% +/- 13% (95% CI, 27% to 38%); and one context, 20% +/- 9% (95% CI, 15% to 24%). Adding context-based queries to full-text queries monotonically improved precision beyond the 0.4 level of recall. Mean improvement was 4.5% at recall 0.5. Adding concept-based search to full-text search improved precision to 19.4% at recall 0.5. CONCLUSIONS: The study demonstrated usefulness of concept-based and context-sensitive queries for enhancing the precision of retrieval from a digital library of semi-structured clinical guideline documents. Concept-based searches outperformed free-text queries, especially when baseline precision was low. In general, the more ontological elements used in the query, the greater the resulting precision.

Detecting computer worms is a highly challenging task. Commonly this task is performed by antivirus software tools that rely on prior explicit knowledge of the worm's code, which is represented by signatures. We present a new approach based on artificial neural networks (ANN) for detecting the presence of computer worms based on the computer's behavioral measures. In order to evaluate the new approach, several computers were infected with seven different worms and more than sixty different parameters of the infected computers were measured. The ANN and two other known classifications techniques, decision tree and k-nearest neighbors, were used to test their ability to classify correctly the presence, and the type, of the computer worms even during heavy user activity on the infected computers. The comparisons between the three approaches suggest that the ANN approach have computational advantages when real-time computation is needed, and has the potential to detect previously unknown worms. In addition, ANN may be used to identify the most relevant, measurable, features and thus reduce the feature dimensionality.

The detection of known and unknown attacks usually requires the interpretation and presentation of very large amounts of time-oriented security data. Using regular means for displaying the data, such as text or tables, is often ineffective. Furthermore, displaying only raw data is not sufficient, because the security expert is still required to derive meaningful conclusions from large amounts of data. In addition, in many cases (e.g., for detecting a virus spreading in the network), an aggregated view of multiple network devices is more effective than a view of each individual device. In this paper we propose an intelligent interface used by a distributed architecture that was described in our previous work, specific to the tasks of knowledge-based interpretation, summarization, query, visualization and interactive exploration of large numbers of time-oriented data. In order to support the interpretation and computation process, we provide automated mechanisms that perform derivation of context-specific, interval-based abstract interpretations (also known as Temporal Abstractions) from raw time-stamped security data, by using a domain-specific knowledge-base (e.g., a period of 5 hours, during the night, of a high number of FTP connections within the context of No User Activity, which might indicate the existence of a Trojan in the computer). The proposed visualization tool includes several functionalities for querying, visualization and exploration of both raw and abstracted time-oriented security data regarding single and multiple network devices.

KNAVE-II is an intelligent interface to a distributed web-based architecture that enables users (e.g., physicians) to query, visualize and explore clinical time-oriented databases. Based on prior studies, we have defined a set of requirements for provision of a service for interactive exploration of time oriented clinical data. The main requirements include the visualization, interactive exploration and explanation of both raw data and multiple levels of concepts abstracted from these data; the exploration of clinical data at different levels of temporal granularity along both absolute (calendar-based) and relative (clinically meaningful) time-lines; the exploration and dynamic visualization of the effects of simulated hypothetical modifications of raw data on the derived concepts; and the provision of generic services (such as statistics, documentation, fast search and retrieval of clinically significant concepts, amongst others). KNAVE-II has been implemented and is currently evaluated by expert clinicians in several medical domains, such as oncology, involving monitoring of chronic patients.

Quality assessment of clinician actions and patient outcomes is a central problem in guideline- or standards-based medical care. In this paper we describe an approach for evaluating and consistently scoring clinician adherence to medical guidelines using the intentions of guideline authors. We present the Quality Indicator Language (QUIL) that may be used to formally specify quality constraints on physician behavior and patient outcomes derived from medical guidelines. We present a modeling and scoring methodology for consistently evaluating multi-step and multi-choice guideline plans based on guideline intentions and their revisions.

We describe a new conceptual methodology and related computational architecture called Knowledge‐based Navigation of Abstractions for Visualization and Explanation (KNAVE). KNAVE is a domain‐independent framework specific to the task of interpretation, summarization, visualization, explanation, and interactive exploration, in a context‐sensitive manner, of time‐oriented raw data and the multiple levels of higher level, interval‐based concepts that can be abstracted from these data. The KNAVE domain‐independent exploration operators are based on the relations defined in the knowledge‐based temporal‐abstraction problem‐solving method, which is used to abstract the data, and thus can directly use the domain‐specific knowledge base on which that method relies. Thus, the domain‐specific semantics are driving the domain‐independent visualization and exploration processes, and the data are viewed through a filter of domain‐specific knowledge. By accessing the domain‐specific temporal‐abstraction knowledge base and the domain‐specific time‐oriented database, the KNAVE modules enable users to query for domain‐specific temporal abstractions and to change the focus of the visualization, thus reusing for a different task (visualization and exploration) the same domain model acquired for abstraction purposes. We focus here on the methodology, but also describe a preliminary evaluation of the KNAVE prototype in a medical domain. Our experiment incorporated seven users, a large medical patient record, and three complex temporal queries, typical of guideline‐based care, that the users were required to answer and/or explore. The results of the preliminary experiment have been encouraging. The new methodology has potentially broad implications for planning, monitoring, explaining, and interactive data mining of time‐oriented data.

Computer-based systems that support health care require large controlled terminologies to manage names and meanings of data elements. These terminologies are not static, because change in health care is inevitable. To share data and applications in health care, we need standards not only for terminologies and concept representation, but also for representing change. To develop a principled approach to managing change, we analyze the requirements of controlled medical terminologies and consider features that frame knowledge-representation systems have to offer. Based on our analysis, we present a concept model, a set of change operations, and a change-documentation model that may be appropriate for controlled terminologies in health care. We are currently implementing our modeling approach within a computational architecture.

OBJECTIVES: The authors discuss the usability of an automated tool that supports entry, by clinical experts, of the knowledge necessary for forming high-level concepts and patterns from raw time-oriented clinical data. DESIGN: Based on their previous work on the RESUME system for forming high-level concepts from raw time-oriented clinical data, the authors designed a graphical knowledge acquisition (KA) tool that acquires the knowledge required by RESUME. This tool was designed using Protégé, a general framework and set of tools for the construction of knowledge-based systems. The usability of the KA tool was evaluated by three expert physicians and three knowledge engineers in three domains-the monitoring of children's growth, the care of patients with diabetes, and protocol-based care in oncology and in experimental therapy for AIDS. The study evaluated the usability of the KA tool for the entry of previously elicited knowledge. MEASUREMENTS: The authors recorded the time required to understand the methodology and the KA tool and to enter the knowledge; they examined the subjects' qualitative comments; and they compared the output abstractions with benchmark abstractions computed from the same data and a version of the same knowledge entered manually by RESUME experts. RESULTS: Understanding RESUME required 6 to 20 hours (median, 15 to 20 hours); learning to use the KA tool required 2 to 6 hours (median, 3 to 4 hours). Entry times for physicians varied by domain-2 to 20 hours for growth monitoring (median, 3 hours), 6 and 12 hours for diabetes care, and 5 to 60 hours for protocol-based care (median, 10 hours). An increase in speed of up to 25 times (median, 3 times) was demonstrated for all participants when the KA process was repeated. On their first attempt at using the tool to enter the knowledge, the knowledge engineers recorded entry times similar to those of the expert physicians' second attempt at entering the same knowledge. In all cases RESUME, using knowledge entered by means of the KA tool, generated abstractions that were almost identical to those generated using the same knowledge entered manually. CONCLUSION: The authors demonstrate that the KA tool is usable and effective for expert physicians and knowledge engineers to enter clinical temporal-abstraction knowledge and that the resulting knowledge bases are as valid as those produced by manual entry.

Temporal interpolation is the task of bridging gaps between time-oriented concepts in a context-sensitive manner. It is a subtask important for solving the temporal-abstraction task-abstraction of interval-based, higher-level concepts from time-stamped data. We present a knowledge-based approach to the temporal-interpolation task and discuss in detail the precise knowledge required by that approach, its theoretical foundations, and the implications of the approach. The temporal-interpolation computational mechanism we discuss relies, among other knowledge types, on a temporal-persistence model. The temporal-persistence model employs local temporal-persistence functions that are temporally bidirectional (i.e. extend a belief measure in a predicate both into the future and into the past) and global, maximal-gap temporal-persistence functions that bridge gaps between interval-based predicates. We investigate the quantitative and qualitative properties implied by both types of persistence functions. We have implemented our approach in the RÉSUMÉ program and evaluated it in several different medical and engineering domains. We discuss the implications of our conceptual and computational methodology for acquisition, maintenance, reuse, and sharing of temporal-abstraction knowledge.

Clinical guidelines can be viewed as generic skeletal-plan schemata that represent clinical procedural knowledge and that are instantiated and refined dynamically by care providers over significant time periods. In the Asgaard project, we are investigating a set of tasks that support the application of clinical guidelines by a care provider other than the guideline's designer. We are focusing on the application of the guideline, recognition of care providers' intentions from their actions, and critique of care providers' actions given the guideline and the patient's medical record. We are developing methods that perform these tasks in multiple clinical domains, given an instance of a properly represented clinical guideline and an electronic medical patient record. In this paper, we point out the precise domain-specific knowledge required by each method, such as the explicit intentions of the guideline designer (represented as temporal patterns to be achieved or avoided). We present a machine-readable language, called Asbru, to represent and to annotate guidelines based on the task-specific ontology. We also introduce an automated tool for the acquisition of clinical guidelines based on the same ontology, developed using the PROTEGE-II framework.

A new domain-independent knowledge-based inference structure is presented, specific to the task of abstracting higher-level concepts from time-stamped data. The framework includes a model of time, parameters, events and contexts. A formal specification of a domain's temporal abstraction knowledge supports acquisition, maintenance, reuse and sharing of that knowledge. The knowledge-based temporal abstraction method decomposes the temporal abstraction task into five subtasks. These subtasks are solved by five domain-independent temporal abstraction mechanisms. The temporal abstraction mechanisms depend on four domain-specific knowledge types: structural, classification (functional), temporal semantic (logical) and temporal dynamic (probabilistic) knowledge. Domain values for all knowledge types are specified when a temporal abstraction system is developed. The knowledge-based temporal abstraction method has been implemented in the RÉSUMÉ system and has been evaluated in several clinical domains (protocol-based care, monitoring of children's growth and therapy of diabetes) and in an engineering domain (monitoring of traffic control), with encouraging results.

We have defined a knowledge-based framework for the creation of abstract, interval-based concepts from time-stamped clinical data, the knowledge-based temporal-abstraction (KBTA) method. The KBTA method decomposes its task into five subtasks; for each subtask we propose a formal solving mechanism. Our framework emphasizes explicit representation of knowledge required for abstraction of time-oriented clinical data, and facilitates its acquisition, maintenance, reuse and sharing. The RESUME system implements the KBTA method. We tested RESUME in several clinical-monitoring domains, including the domain of monitoring patients who have insulin-dependent diabetes. We acquired from a diabetes-therapy expert diabetes-therapy temporal-abstraction knowledge. Two diabetes-therapy experts (including the first one) created temporal abstractions from about 800 points of diabetic-patients' data. RESUME generated about 80% of the abstractions agreed by both experts; about 97% of the generated abstractions were valid. We discuss the advantages and limitations of the current architecture.

Provision of automated support for planning protocol-directed therapy requires a computer program to take as input clinical data stored in an electronic patient-record system and to generate as output recommendations for therapeutic interventions and laboratory testing that are defined by applicable protocols. This paper presents a synthesis of research carried out at Stanford University to model the therapy-planning task and to demonstrate a component-based architecture for building protocol-based decision-support systems. We have constructed general-purpose software components that (1) interpret abstract protocol specifications to construct appropriate patient-specific treatment plans; (2) infer from time-stamped patient data higher-level, interval-based, abstract concepts; (3) perform time-oriented queries on a time-oriented patient database; and (4) allow acquisition and maintenance of protocol knowledge in a manner that facilitates efficient processing both by humans and by computers. We have implemented these components in a computer system known as EON. Each of the components has been developed, evaluated, and reported independently. We have evaluated the integration of the components as a composite architecture by implementing T-HELPER, a computer-based patient-record system that uses EON to offer advice regarding the management of patients who are following clinical trial protocols for AIDS or HIV infection. A test of the reuse of the software components in a different clinical domain demonstrated rapid development of a prototype application to support protocol-based care of patients who have breast cancer.

Problem-solving methods for knowledge-based systems establish the behavior of such systems by defining the roles in which domain knowledge is used and the ordering of inferences. Developers can compose problem-solving methods that accomplish complex application tasks from primitive, reusable methods. The key steps in this development approach are task analysis, method selection (from a library), and method configuration. Protégé-ii is a knowledge-engineering environment that allows developers to select and configure problem-solving methods. In addition, Protégé-ii generates domain-specific knowledge-acquisition tools that domain specialists can use to create knowledge bases on which the methods may operate. The board-game method is a problem-solving method that defines control knowledge for a class of tasks that developers can model in a highly specific way. The method adopts a conceptual model of problem solving in which the solution space is construed as a “game board” on which the problem solver moves “playing pieces” according to prespecified rules. This familiar conceptual model simplifies the developer's cognitive demands when configuring the board-game method to support new application tasks. We compare configuration of the board-game method to that of a chronological-backtracking problem-solving method for the same application tasks (for example, towers of Hanoi and the Sisyphus room-assignment problem). We also examine how method designers can specialize problem-solving methods by making ontological commitments to certain classes of tasks. We exemplify this technique by specializing the chronological-backtracking method to the board-game method.