How to deliver best care in various clinical settings remains a vexing problem. All pertinent healthcare-related questions have not, cannot, and will not be addressable with costly time- and resource-consuming controlled clinical trials. At present, evidence-based guidelines can address only a small fraction of the types of care that clinicians deliver. Furthermore, underserved areas rarely can access state-of-the-art evidence-based guidelines in real-time, and often lack the wherewithal to implement advanced guidelines. Care providers in such settings frequently do not have sufficient training to undertake advanced guideline implementation. Nevertheless, in advanced modern healthcare delivery environments, use of eActions (validated clinical decision support systems) could help overcome the cognitive limitations of overburdened clinicians. Widespread use of eActions will require surmounting current healthcare technical and cultural barriers and installing clinical evidence/data curation systems. The authors expect that increased numbers of evidence-based guidelines will result from future comparative effectiveness clinical research carried out during routine healthcare delivery within learning healthcare systems.

Clinical decision-making is based on knowledge, expertise, and authority, with clinicians approving almost every intervention-the starting point for delivery of "All the right care, but only the right care," an unachieved healthcare quality improvement goal. Unaided clinicians suffer from human cognitive limitations and biases when decisions are based only on their training, expertise, and experience. Electronic health records (EHRs) could improve healthcare with robust decision-support tools that reduce unwarranted variation of clinician decisions and actions. Current EHRs, focused on results review, documentation, and accounting, are awkward, time-consuming, and contribute to clinician stress and burnout. Decision-support tools could reduce clinician burden and enable replicable clinician decisions and actions that personalize patient care. Most current clinical decision-support tools or aids lack detail and neither reduce burden nor enable replicable actions. Clinicians must provide subjective interpretation and missing logic, thus introducing personal biases and mindless, unwarranted, variation from evidence-based practice. Replicability occurs when different clinicians, with the same patient information and context, come to the same decision and action. We propose a feasible subset of therapeutic decision-support tools based on credible clinical outcome evidence: computer protocols leading to replicable clinician actions (eActions). eActions enable different clinicians to make consistent decisions and actions when faced with the same patient input data. eActions embrace good everyday decision-making informed by evidence, experience, EHR data, and individual patient status. eActions can reduce unwarranted variation, increase quality of clinical care and research, reduce EHR noise, and could enable a learning healthcare system.

OBJECTIVE: To facilitate the development of standards-based clinical decision support (CDS) systems, we review the current set of CDS standards that are based on Health Level Seven International Fast Healthcare Interoperability Resources (FHIR). Widespread adoption of these standards may help reduce healthcare variability, improve healthcare quality, and improve patient safety. TARGET AUDIENCE: This tutorial is designed for the broad informatics community, some of whom may be unfamiliar with the current, FHIR-based CDS standards. SCOPE: This tutorial covers the following standards: Arden Syntax (using FHIR as the data model), Clinical Quality Language, FHIR Clinical Reasoning, SMART on FHIR, and CDS Hooks. Detailed descriptions and selected examples are provided.

OBJECTIVE: Develop an efficient non-clinical method for identifying promising computer-based protocols for clinical study. An in silico comparison can provide information that informs the decision to proceed to a clinical trial. The authors compared two existing computer-based insulin infusion protocols: eProtocol-insulin from Utah, USA, and Glucosafe from Denmark. MATERIALS AND METHODS: The authors used eProtocol-insulin to manage intensive care unit (ICU) hyperglycemia with intravenous (IV) insulin from 2004 to 2010. Recommendations accepted by the bedside clinicians directly link the subsequent blood glucose values to eProtocol-insulin recommendations and provide a unique clinical database. The authors retrospectively compared in silico 18,984 eProtocol-insulin continuous IV insulin infusion rate recommendations from 408 ICU patients with those of Glucosafe, the candidate computer-based protocol. The subsequent blood glucose measurement value (low, on target, high) was used to identify if the insulin recommendation was too high, on target, or too low. RESULTS: Glucosafe consistently provided more favorable continuous IV insulin infusion rate recommendations than eProtocol-insulin for on target (64% of comparisons), low (80% of comparisons), or high (70% of comparisons) blood glucose. Aggregated eProtocol-insulin and Glucosafe continuous IV insulin infusion rates were clinically similar though statistically significantly different (Wilcoxon signed rank test P = .01). In contrast, when stratified by low, on target, or high subsequent blood glucose measurement, insulin infusion rates from eProtocol-insulin and Glucosafe were statistically significantly different (Wilcoxon signed rank test, P < .001), and clinically different. DISCUSSION: This in silico comparison appears to be an efficient nonclinical method for identifying promising computer-based protocols. CONCLUSION: Preclinical in silico comparison analytical framework allows rapid and inexpensive identification of computer-based protocol care strategies that justify expensive and burdensome clinical trials.

OBJECTIVE: Natural language processing (NLP) tasks are commonly decomposed into subtasks, chained together to form processing pipelines. The residual error produced in these subtasks propagates, adversely affecting the end objectives. Limited availability of annotated clinical data remains a barrier to reaching state-of-the-art operating characteristics using statistically based NLP tools in the clinical domain. Here we explore the unique linguistic constructions of clinical texts and demonstrate the loss in operating characteristics when out-of-the-box part-of-speech (POS) tagging tools are applied to the clinical domain. We test a domain adaptation approach integrating a novel lexical-generation probability rule used in a transformation-based learner to boost POS performance on clinical narratives. METHODS: Two target corpora from independent healthcare institutions were constructed from high frequency clinical narratives. Four leading POS taggers with their out-of-the-box models trained from general English and biomedical abstracts were evaluated against these clinical corpora. A high performing domain adaptation method, Easy Adapt, was compared to our newly proposed method ClinAdapt. RESULTS: The evaluated POS taggers drop in accuracy by 8.5-15% when tested on clinical narratives. The highest performing tagger reports an accuracy of 88.6%. Domain adaptation with Easy Adapt reports accuracies of 88.3-91.0% on clinical texts. ClinAdapt reports 93.2-93.9%. CONCLUSIONS: ClinAdapt successfully boosts POS tagging performance through domain adaptation requiring a modest amount of annotated clinical data. Improving the performance of critical NLP subtasks is expected to reduce pipeline error propagation leading to better overall results on complex processing tasks.

The Arden Syntax is an HL7 standard language for representing medical knowledge as logic statements. Despite nearly 2 decades of availability, Arden Syntax has not been widely used. This has been attributed to the lack of a generally available compiler to implement the logic, to Arden's complex syntax, to the challenges of mapping local data to data references in the Medical Logic Modules (MLMs), or, more globally, to the general absence of decision support in healthcare computing. An XML representation (ArdenML) may partially address the technical challenges. MLMs created in ArdenML can be converted into executable files using standard transforms written in the Extensible Stylesheet Language Transformation (XSLT) language. As an example, we have demonstrated an approach to executing MLMs written in ArdenML using the Drools business rule management system. Extensions to ArdenML make it possible to generate a user interface through which an MLM developer can test for logical errors.

OBJECTIVE: To understand how the source of information affects different adverse event (AE) surveillance methods. DESIGN: Retrospective analysis of inpatient adverse drug events (ADEs) and hospital-associated infections (HAIs) detected by either a computerized surveillance system (CSS) or manual chart review (MCR). MEASUREMENT: Descriptive analysis of events detected using the two methods by type of AE, type of information about the AE, and sources of the information. RESULTS: CSS detected more HAIs than MCR (92% vs 34%); however, a similar number of ADEs was detected by both systems (52% vs 51%). The agreement between systems was greater for HAIs than ADEs (26% vs 3%). The CSS missed events that did not have information in coded format or that were described only in physician narratives. The MCR detected events missed by CSS using information in physician narratives. Discharge summaries were more likely to contain information about AEs than any other type of physician narrative, followed by emergency department reports for HAIs and general consult notes for ADEs. Some ADEs found by MCR were detected by CSS but not verified by a clinician. LIMITATIONS: Inability to distinguish between CSS false positives and suspected AEs for cases in which the clinician did not document their assessment in the CSS. CONCLUSION: The effect that information source has on different surveillance methods depends on the type of AE. Integrating information from physician narratives with CSS using natural language processing would improve the detection of ADEs more than HAIs.

OBJECTIVE: Infobuttons are decision support tools that provide links within electronic medical record systems to relevant content in online information resources. The aim of infobuttons is to help clinicians promptly meet their information needs. The objective of this study was to determine whether infobutton links that direct to specific content topics ("topic links") are more effective than links that point to general overview content ("nonspecific links"). DESIGN: Randomized controlled trial with a control and an intervention group. Clinicians in the control group had access to nonspecific links, while those in the intervention group had access to topic links. MEASUREMENTS: Infobutton session duration, number of infobutton sessions, session success rate, and the self-reported impact that the infobutton session produced on decision making. RESULTS: The analysis was performed on 90 subjects and 3,729 infobutton sessions. Subjects in the intervention group spent 17.4% less time seeking for information (35.5 seconds vs. 43 seconds, p = 0.008) than those in the control group. Subjects in the intervention group used infobuttons 20.5% (22 sessions vs. 17.5 sessions, p = 0.21) more often than in the control group, but the difference was not significant. The information seeking success rate was equally high in both groups (89.4% control vs. 87.2% intervention, p = 0.99). Subjects reported a high positive clinical impact (i.e., decision enhancement or knowledge update) in 62% of the sessions. Limitations The exclusion of users with a low frequency of infobutton use and the focus on medication-related information needs may limit the generalization of the results. The session outcomes measurement was based on clinicians' self-assessment and therefore prone to bias. CONCLUSION: The results support the hypothesis that topic links are more efficient than nonspecific links regarding the time seeking for information. It is unclear whether the statistical difference demonstrated will result in a clinically significant impact. However, the overall results confirm previous evidence that infobuttons are effective at helping clinicians to answer questions at the point of care and demonstrate a modest incremental change in the efficiency of information delivery for routine users of this tool.

OBJECTIVE: Do-not-resuscitate (DNR) orders and advance directives are increasingly prevalent and may affect medical interventions and outcomes. Simple, automated techniques to identify patients with DNR orders do not currently exist but could help avoid costly and time-consuming chart review. This study hypothesized that a decision to withhold cardiopulmonary resuscitation would be included in a patient's dictated reports. The authors developed and validated a simple computerized search method, which screens dictated reports to detect patients with DNR status. METHODS: A list of concepts related to DNR order documentation was developed using emergency department, hospital admission, consult, and hospital discharge reports of 665 consecutive, hospitalized pneumonia patients during a four-year period (1995-1999). The list was validated in an independent group of 190 consecutive inpatients with pneumonia during a five-month period (1999-2000). The reference standard for the presence of DNR orders was manual chart review of all study patients. Sensitivity, specificity, predictive values, and nonerror rates were calculated for individual and combined concepts. RESULTS: The list of concepts included: DNR, Do Not Attempt to Resuscitate (DNAR), DNI, NCR, advanced directive, living will, power of attorney, Cardiopulmonary Resuscitation (CPR), defibrillation, arrest, resuscitate, code, and comfort care. As determined by manual chart review, a DNR order was written for 32.6% of patients in the derivation and for 31.6% in the validation group. Dictated reports included DNR order-related information for 74.5% of patients in the derivation and 73% in the validation group. If mentioned in the dictated report, the combined keyword search had a sensitivity of 74.2% in the derivation group (70.0% in the validation group), a specificity of 91.5% (81.5%), a positive predictive value of 80.9% (63.6%), a negative predictive value of 88.0% (85.5%), and a nonerror rate of 85.9% (77.9%). DNR and resuscitate were the most frequently used and power of attorney and advanced directives the least frequently used terms. CONCLUSION: Dictated hospital reports frequently contained DNR order-related information for patients with a written DNR order. Using an uncomplicated keyword search, electronic screening of dictated reports yielded good accuracy for identifying patients with DNR order information.

Planning the clinical evaluation of a computerized decision support system requires a strategy that encompasses the different aspects of the clinical problem, the technical difficulties of software and hardware integration and implementation, the behavioral aspects of the targeted users, and the discipline of study design. Although clinical information systems are becoming more widely available, only a few decision support systems have been formally evaluated in clinical environments. Published accounts of difficulties associated with the clinical evaluation of decision support systems remain scarce. The authors report on a variety of behavioral, logistical, technical, clinical, cost, and work flow issues that they had to address when choosing a study design for a clinical trial for the evaluation of an integrated, real-time decision support system for the automatic identification of patients likely to have pneumonia in an emergency department. In the absence of a true gold standard, they show how they created a credible, clinically acceptable, and economical reference standard for the diagnosis of pneumonia, to determine the overall accuracy of the system. For the creation of a reference standard, they describe the importance of recognizing verification bias and avoiding it. Finally, advantages and disadvantages of different study designs are explored with respect to the targeted users and the clinical setting.

OBJECTIVE: This study examined whether clinical data routinely available in a computerized patient record (CPR) can be used to drive a complex guideline that supports physicians in real time and at the point of care in assessing the risk of mortality for patients with community-acquired pneumonia. SETTING: Emergency department of a tertiary-care hospital. DESIGN: Retrospective analysis with medical chart review. PATIENTS: All 241 inpatients during a 17-month period (Jun 1995 to Nov 1996) who presented to the emergency department and had a primary discharge diagnosis of community-acquired pneumonia. METHODS/MAIN OUTCOME MEASURES: The 20 guideline variables were extracted from the CPR (HELP System) and the paper chart. The risk score and the risk class of the Pneumonia Severity Index were computed using data from the CPR alone and from a reference standard of all data available in the paper chart and the CPR at the time of the emergency department encounters. Availability and concordance were quantified to determine data quality. The type and cause of errors were analyzed depending on the source and format of the clinical variables. RESULTS: Of the 20 guideline variables, 12 variables were required to be present for every computer-charted emergency department patient, seven variables were required for selected patients only, and one variable was not typically available in the HELP System during a patient's encounter. The risk class was identical for 86.7 percent of the patients. The majority of patients with different risk classes were assigned too low a risk class. The risk scores were identical for 72.1 percent of the patients. The average availability was 0.99 for the data elements that were required to be present and 0.79 for the data elements that were not required to be present. The average concordance was 0.98 when all a patient's variables were taken into account. The cause of error was attributed to the nurse charting in 77 percent of the cases and to the computerized evaluation in 23 percent. The type of error originated from the free-text fields in 64 percent, from coded fields in 21 percent, from vital signs in 14 percent, and from laboratory results in 1 percent. CONCLUSION: From a clinical perspective, the current level of data quality in the HELP System supports the automation and the prospective evaluation of the Pneumonia Severity Index as a computerized decision support tool.

OBJECTIVE: To evaluate the performance of a natural language processing system in extracting pneumonia-related concepts from chest x-ray reports. DESIGN: Four physicians, three lay persons, a natural language processing system, and two keyword searches (designated AAKS and KS) detected the presence or absence of three pneumonia-related concepts and inferred the presence or absence of acute bacterial pneumonia from 292 chest x-ray reports. Gold standard: Majority vote of three independent physicians. Reliability of the gold standard was measured. OUTCOME MEASURES: Recall, precision, specificity, and agreement (using Finn's R: statistic) with respect to the gold standard. Differences between the physicians and the other subjects were tested using the McNemar test for each pneumonia concept and for the disease inference of acute bacterial pneumonia. RESULTS: Reliability of the reference standard ranged from 0.86 to 0.96. Recall, precision, specificity, and agreement (Finn R:) for the inference on acute bacterial pneumonia were, respectively, 0.94, 0.87, 0.91, and 0.84 for physicians; 0.95, 0.78, 0.85, and 0.75 for natural language processing system; 0.46, 0.89, 0.95, and 0.54 for lay persons; 0.79, 0.63, 0.71, and 0.49 for AAKS; and 0.87, 0.70, 0.77, and 0.62 for KS. The McNemar pairwise comparisons showed differences between one physician and the natural language processing system for the infiltrate concept and between another physician and the natural language processing system for the inference on acute bacterial pneumonia. The comparisons also showed that most physicians were significantly different from the other subjects in all pneumonia concepts and the disease inference. CONCLUSION: In extracting pneumonia related concepts from chest x-ray reports, the performance of the natural language processing system was similar to that of physicians and better than that of lay persons and keyword searches. The encoded pneumonia information has the potential to support several pneumonia-related applications used in our institution. The applications include a decision support system called the antibiotic assistant, a computerized clinical protocol for pneumonia, and a quality assurance application in the radiology department.

OBJECTIVE: To report lessons learned from evaluation of an automated interface between a hospital clinical information system and a severity of illness index. DESIGN: A system was developed to convert coded electronic patient findings from the HELP System at LDS Hospital into the attributes used by the Computerized Severity Index (CSI) to calculate a severity of illness score. Performance was assessed by comparing the automated CSI score with the manual CSI score (from paper chart review) and by evaluating changes introduced by augmenting the manual CSI score with verified patient data discovered by the automated CSI method. MEASUREMENTS: The strengths and weaknesses of each method are presented. RESULTS: The automated CSI score matched the manual CSI score in 61% of the cases. Sources of errors were analyzed. When the automated score was in error, two-thirds of the time it was due to the lack of codes in the HELP system representing CSI concepts; one-third of the time it was due to nurses not using established HELP system codes. Surprisingly, significant problems were also discovered in the manual system, making it difficult to define a "gold standard". CONCLUSIONS: Automated computerized severity indices have great potential for future applicability once their performance exceeds that of the time-consuming manual chart review method. Neither automated nor manual methods are adequate at the present time. This area remains a fertile ground for future research.

OBJECTIVE: Develop a model for structured and encoded representation of medical information that supports human review, decision support applications, ad hoc queries, statistical analysis, and natural-language processing. DESIGN: A medical information representation model was developed from manual and semiautomated analysis of patient data. The key assumption of the model is that medical information can be represented as a series of linked events. The event representation has two main components. The first component is a frame or template definition that specifies the attributes of the event. The second component is a structured vocabulary, the terms of which are taken as the values of the slots in the event template structure. Individual event instances are linked by specific named relationships. RESULTS: The proposed model was used to represent a chest-radiograph report. CONCLUSIONS: The event model of medical information representation provides a mechanism for formal definition of the logical structure of medical data and allows explicit time-oriented and associative relationships between event instances.