OBJECTIVE: The Greater Plains Collaborative (GPC) and other PCORnet Clinical Data Research Networks capture healthcare utilization within their health systems. Here, we describe a reusable environment (GPC Reusable Observable Unified Study Environment [GROUSE]) that integrates hospital and electronic health records (EHRs) data with state-wide Medicare and Medicaid claims and assess how claims and clinical data complement each other to identify obesity and related comorbidities in a patient sample. MATERIALS AND METHODS: EHR, billing, and tumor registry data from 7 healthcare systems were integrated with Center for Medicare (2011-2016) and Medicaid (2011-2012) services insurance claims to create deidentified databases in Informatics for Integrating Biology & the Bedside and PCORnet Common Data Model formats. We describe technical details of how this federally compliant, cloud-based data environment was built. As a use case, trends in obesity rates for different age groups are reported, along with the relative contribution of claims and EHR data-to-data completeness and detecting common comorbidities. RESULTS: GROUSE contained 73 billion observations from 24 million unique patients (12.9 million Medicare; 13.9 million Medicaid; 6.6 million GPC patients) with 1 674 134 patients crosswalked and 983 450 patients with body mass index (BMI) linked to claims. Diagnosis codes from EHR and claims sources underreport obesity by 2.56 times compared with body mass index measures. However, common comorbidities such as diabetes and sleep apnea diagnoses were more often available from claims diagnoses codes (1.6 and 1.4 times, respectively). CONCLUSION: GROUSE provides a unified EHR-claims environment to address health system and federal privacy concerns, which enables investigators to generalize analyses across health systems integrated with multistate insurance claims.

Objective: Diabetic kidney disease (DKD) is one of the most frequent complications in diabetes associated with substantial morbidity and mortality. To accelerate DKD risk factor discovery, we present an ensemble feature selection approach to identify a robust set of discriminant factors using electronic medical records (EMRs). Material and Methods: We identified a retrospective cohort of 15 645 adult patients with type 2 diabetes, excluding those with pre-existing kidney disease, and utilized all available clinical data types in modeling. We compared 3 machine-learning-based embedded feature selection methods in conjunction with 6 feature ensemble techniques for selecting top-ranked features in terms of robustness to data perturbations and predictability for DKD onset. Results: The gradient boosting machine (GBM) with weighted mean rank feature ensemble technique achieved the best performance with an AUC of 0.82 [95%-CI, 0.81-0.83] on internal validation and 0.71 [95%-CI, 0.68-0.73] on external temporal validation. The ensemble model identified a set of 440 features from 84 872 unique clinical features that are both predicative of DKD onset and robust against data perturbations, including 191 labs, 51 visit details (mainly vital signs), 39 medications, 34 orders, 30 diagnoses, and 95 other clinical features. Discussion: Many of the top-ranked features have not been included in the state-of-art DKD prediction models, but their relationships with kidney function have been suggested in existing literature. Conclusion: Our ensemble feature selection framework provides an option for identifying a robust and parsimonious feature set unbiasedly from EMR data, which effectively aids in knowledge discovery for DKD risk factors.

The Greater Plains Collaborative (GPC) is composed of 10 leading medical centers repurposing the research programs and informatics infrastructures developed through Clinical and Translational Science Award initiatives. Partners are the University of Kansas Medical Center, Children's Mercy Hospital, University of Iowa Healthcare, the University of Wisconsin-Madison, the Medical College of Wisconsin and Marshfield Clinic, the University of Minnesota Academic Health Center, the University of Nebraska Medical Center, the University of Texas Health Sciences Center at San Antonio, and the University of Texas Southwestern Medical Center. The GPC network brings together a diverse population of 10 million people across 1300 miles covering seven states with a combined area of 679 159 square miles. Using input from community members, breast cancer was selected as a focus for cohort building activities. In addition to a high-prevalence disorder, we also selected a rare disease, amyotrophic lateral sclerosis.

OBJECTIVE: Alerting systems, a type of clinical decision support, are increasingly prevalent in healthcare, yet few studies have concurrently measured the appropriateness of alerts with provider responses to alerts. Recent reports of suboptimal alert system design and implementation highlight the need for better evaluation to inform future designs. The authors present a comprehensive framework for evaluating the clinical appropriateness of synchronous, interruptive medication safety alerts. METHODS: Through literature review and iterative testing, metrics were developed that describe successes, justifiable overrides, provider non-adherence, and unintended adverse consequences of clinical decision support alerts. The framework was validated by applying it to a medication alerting system for patients with acute kidney injury (AKI). RESULTS: Through expert review, the framework assesses each alert episode for appropriateness of the alert display and the necessity and urgency of a clinical response. Primary outcomes of the framework include the false positive alert rate, alert override rate, provider non-adherence rate, and rate of provider response appropriateness. Application of the framework to evaluate an existing AKI medication alerting system provided a more complete understanding of the process outcomes measured in the AKI medication alerting system. The authors confirmed that previous alerts and provider responses were most often appropriate. CONCLUSION: The new evaluation model offers a potentially effective method for assessing the clinical appropriateness of synchronous interruptive medication alerts prior to evaluating patient outcomes in a comparative trial. More work can determine the generalizability of the framework for use in other settings and other alert types.

OBJECTIVE: To determine characteristics and effects of nurse dosing over-rides of a clinical decision support system (CDSS) for intensive insulin therapy (IIT) in critical care units. DESIGN: Retrospective analysis of patient database records and ethnographic study of nurses using IIT CDSS. MEASUREMENTS: The authors determined the frequency, direction-greater than recommended (GTR) and less than recommended (LTR)- and magnitude of over-rides, and then compared recommended and over-ride doses' blood glucose (BG) variability and insulin resistance, two measures of IIT CDSS associated with mortality. The authors hypothesized that rates of hypoglycemia and hyperglycemia would be greater for recommended than over-ride doses. Finally, the authors observed and interviewed nurse users. RESULTS: 5.1% (9075) of 179,452 IIT CDSS doses were over-rides. 83.4% of over-ride doses were LTR, and 45.5% of these were ≥ 50% lower than recommended. In contrast, 78.9% of GTR doses were ≤ 25% higher than recommended. When recommended doses were administered, the rate of hypoglycemia was higher than the rate for GTR (p = 0.257) and LTR (p = 0.033) doses. When recommended doses were administered, the rate of hyperglycemia was lower than the rate for GTR (p = 0.003) and LTR (p < 0.001) doses. Estimates of patients' insulin requirements were higher for LTR doses than recommended and GTR doses. Nurses reported trusting IIT CDSS overall but appeared concerned about recommendations when administering LTR doses. CONCLUSION: When over-riding IIT CDSS recommendations, nurses overwhelmingly administered LTR doses, which emphasized prevention of hypoglycemia but interfered with hyperglycemia control, especially when BG was >150 mg/dl. Nurses appeared to consider the amount of a recommended insulin dose, not a patient's trend of insulin resistance, when administering LTR doses overall. Over-rides affected IIT CDSS protocol performance.

Medication information is one of the most important types of clinical data in electronic medical records. It is critical for healthcare safety and quality, as well as for clinical research that uses electronic medical record data. However, medication data are often recorded in clinical notes as free-text. As such, they are not accessible to other computerized applications that rely on coded data. We describe a new natural language processing system (MedEx), which extracts medication information from clinical notes. MedEx was initially developed using discharge summaries. An evaluation using a data set of 50 discharge summaries showed it performed well on identifying not only drug names (F-measure 93.2%), but also signature information, such as strength, route, and frequency, with F-measures of 94.5%, 93.9%, and 96.0% respectively. We then applied MedEx unchanged to outpatient clinic visit notes. It performed similarly with F-measures over 90% on a set of 25 clinic visit notes.

OBJECTIVE: Hyperglycemia worsens clinical outcomes in critically ill patients. Precise glycemia control using intravenous insulin improves outcomes. To determine if we could improve glycemia control over a previous paper-based, manual protocol, authors implemented, in a surgical intensive care unit (SICU), an intravenous insulin protocol integrated into a care provider order entry (CPOE) system. DESIGN: Retrospective before-after study of consecutive adult patients admitted to a SICU during pre (manual protocol, 32 days) and post (computer-based protocol, 49 days) periods. MEASUREMENTS: Percentage of glucose readings in ideal range of 70-109 mg/dl, and minutes spent in ideal range of control during the first 5 days of SICU stay. RESULTS: The computer-based protocol reduced time from first glucose measurement to initiation of insulin protocol, improved the percentage of all SICU glucose readings in the ideal range, and improved control in patients on IV insulin for > or =24 hours. Hypoglycemia (<40 mg/dl) was rare in both groups. CONCLUSION: The CPOE-based intravenous insulin protocol improved glycemia control in SICU patients compared to a previous manual protocol, and reduced time to insulin therapy initiation. Integrating a computer-based insulin protocol into a CPOE system achieved efficient, safe, and effective glycemia control in SICU patients.

Background Up to 38% of inpatient medication errors occur at the administration stage. Although they reduce prescribing errors, computerized provider order entry (CPOE) systems do not prevent administration errors or timing discrepancies. This study determined the degree to which CPOE medication orders matched actual dose administration times. METHODS At a 658-bed academic hospital with CPOE but lacking electronic medication administration charting, authors randomly selected adult patients with eligible medication orders from historical 1999-2003 CPOE log files. Retrospective manual chart audits compared expected (from CPOE) and actual timing of medication administrations. Outcomes included: dose omissions, median lag times between ordered and charted administrations, unauthorized doses, wrong dose errors, and the rate of nurses' medication schedule shifting. RESULTS Dose omissions occurred in 756 of 6019 (12.6%) audited administration opportunities; only 313 of the omissions (5.2% of opportunities) were unexplained. Wrong doses and unexpected doses occurred for 0.1% and 0.7% of opportunities, respectively. Median lag from expected first dose to actual charted administration time was 27 minutes (IQR 0-127). Nursing staff shifted from ordered to alternate administration schedules for 10.7% of regularly scheduled recurring medication orders. Chart review identified reasons for dose omissions, delays, and dose shifting. CONCLUSION Inpatient CPOE orders are legible and conveyed electronically to nurses and the pharmacy. Nonetheless, ward-based medication administrations do not consistently occur as ordered. Medication administration discrepancies are likely to persist even after implementing CPOE and bar-coded medication administration unless recommended interventions are made to address issues such as determining the true urgency of medication administration, avoiding overlapping duplicative medication orders, and developing a safe means for shifting dosing schedules.

OBJECTIVE: In the context of an inpatient care provider order entry (CPOE) system, to evaluate the impact of a decision support tool on integration of cardiology "best of care" order sets into clinicians' admission workflow, and on quality measures for the management of acute myocardial infarction (AMI) patients. DESIGN: A before-and-after study of physician orders evaluated (1) per-patient use rates of standardized acute coronary syndrome (ACS) order set and (2) patient-level compliance with two individual recommendations: early aspirin ordering and beta-blocker ordering. MEASUREMENTS: The effectiveness of the intervention was evaluated for (1) all patients with ACS (suspected for AMI at the time of admission) (N = 540) and (2) the subset of the ACS patients with confirmed discharge diagnosis of AMI (n = 180) who comprise the recommended target population who should receive aspirin and/or beta-blockers. Compliance rates for use of the ACS order set, aspirin ordering, and beta-blocker ordering were calculated as the percentages of patients who had each action performed within 24 hours of admission. RESULTS: For all ACS admissions, the decision support tool significantly increased use of the ACS order set (p = 0.009). Use of the ACS order set led, within the first 24 hours of hospitalization, to a significant increase in the number of patients who received aspirin (p = 0.001) and a nonsignificant increase in the number of patients who received beta-blockers (p = 0.07). Results for confirmed AMI cases demonstrated similar increases, but did not reach statistical significance. CONCLUSION: The decision support tool increased optional use of the ACS order set, but room for additional improvement exists.

We commend Shiffman and colleagues (“Bridging the Guideline Implementation Gap: A Systematic, Document-Centered Approach to Guideline Implementation”1) for highlighting the challenges of integrating guidelines into clinical practice and proposing pragmatic mechanisms for addressing them. We note, however, that the approach advocated by Shiffman et al., as well as by numerous other groups recently,2–8 is fundamentally a document-centric model. This approach may lead others to assume that representing a guideline correctly as a “computer-readable” document is the majority of the work required for implementation success. Although the “understanding” and representation of the clinical content of a guideline are a sine qua non for its local implementation, the document-centric approach leaves a substantial gap between the idealized document model and any specific guideline implementation in a local clinical system. This considerable gap is not unlike the “curly braces” problem documented for the Arden Syntax a decade ago.3–5 We estimate that 90% of the effort required for successful guideline implementation is (and must be) local, and the remaining 10% of the effort involves “getting the document right.” We believe that an alternative approach to local guideline implementation is to focus on the guideline's recommended actions; on the capabilities of the local care provider order entry (CPOE) or electronic health record (EHR) system that will serve as the “effector mechanism” for the guideline; on locally available computational and clinical resources; and on the guideline's required “clinical infrastructure.” We believe that guidelines should be implemented locally and directly (with a systematic approach, as described below) via local clinical systems (as opposed to a quasi-automatic implementation using a computer-readable, nationally disseminated document). The goal of both the “document-centric” and the “locally customized and guided” approaches is the same: implementation of locally effective guidelines that appropriately influence clinical decision making, resulting in desirable actions that improve patient outcomes. Local guideline implementation requires the following understanding, resources, and efforts: A local, clinically expert champion (or group of champions) who will customize the national guideline to be compatible with local capabilities and practices and, more importantly, take ownership of guideline evolution locally over time. While national-level guidelines should form the basis of evidence-based practice, the unfortunate truth is that national guideline developers rarely reconvene to systematically update guidelines in a timely manner. Unless local experts take responsibility for guideline implementation in the present, and for future updates, the institution where a national guideline is implemented over time becomes at risk of practicing “the ‘perfect’ medicine of bygone eras” (e.g., national standards from 5 to 10 years ago). A locally developed consensus among clinicians across services on how to implement each guideline (e.g., across Medicine, Surgery, Obstetrics/Gynecology, Pediatrics, Emergency Department). This may include modifying the national guideline in various ways: Guideline distillation and presentation. What are the resulting local actions (orders) of the guideline? After focusing on the orders, determine whether the conditional statements qualifying the orders lend themselves to explanatory text or whether a more sophisticated “advisor” program with complex calculations is required. The choices may result in implementation of a simple guideline (e.g., evidence-based “pick-lists” specifying care options for a patient admitted with acute coronary syndrome) as an “order set.”9 Order sets for the most part leave “branching logic” choices up to the end user by providing textual instructions over each subsection of orderables (e.g., “Select one of the following beta-blockers that is most appropriate for the patient from the list below.”). In contrast, another, more complex guideline might require implementation as a programcode–based algorithm (“advisor”) that calculates patient-specific doses and recommendations based on known patient demographics, clinical parameters (e.g., renal function, current orders, clinical diagnoses, weight, height), and laboratory results (e.g., coagulation studies, renal function tests, serological results). For example, Web-based advisors are helpful for ordering multicomponent total parenteral nutrition in a neonatal intensive care unit, which requires careful balancing of electrolytes, fluids, caloric sources, and involves many patient-specific “rules.” Web-based advisors are also helpful for combining complex textual instructions with patient-specific calculations such as for antibiotic selection and dosing based on clinical indications. Note, however, that hardware and software issues locally may influence guideline implementation choices. An old MS-DOS® character-based interface cannot support the bandwidth for complex advisors that a Web-based, multitiered architecture can support, but the former may be at least as good for implementing simple pick-lists. Guideline interpretation/translation locally. How are radiology procedures and the pharmacy formulary represented locally in contrast to the text in the guideline? What orderables inferred in the guideline can actually be ordered in the local clinical system? (e.g., if Doppler studies of the legs are suggested in a guideline for “diagnosis and treatment of suspected deep venous thrombosis,” does the local system allow unilateral or bilateral Doppler ordering and in a manner that is consistent with guideline recommendations?) Do new orderables need to be added or will comments or additional parameters within existing orders suffice to fulfill guideline requirements? Guideline creation based on informed decision making about optimal local methods. Having decided on the appropriate orderables (actions) and whether to create the guideline as an order set or an advisor, what additional details must be specified? If the guideline is best implemented via order sets, should there be a single order set or several linked (nested) order sets (e.g., a main order set and another order set for medications that might already have been created for a different purpose)? If the guideline is more complex, and a multifaceted “advisor” is necessary, can the advisor be simple and one pass (e.g., total parenteral nutrition ordering in a neonatal intensive care unit) or will it involve multistage “component” advisors? For example, for an “anticoagulation advisor,” the initial phase might consist of ordering deep venous thrombosis prophylaxis for “normal” patients at bed rest. The next phase might involve ordering the correct diagnostic procedures, baseline laboratory tests, and “coverage” anticoagulants for patients with “suspected deep venous thrombosis.” The next phase should advise users how to initiate “definitive therapy” per national guidelines once a diagnosis of “deep venous thrombosis” is firmly established. Finally, the last phase might consist of adjusting heparin (or low molecular weight heparin) doses per national guidelines after initial therapy was ordered, based on follow-up information (such as activated partial thromboplastin test results for monitoring heparin dosing). Such multistage protocols must “recognize” the previous state of the patient in the sequence of the protocol, as well as be able to “trigger” the next step on appropriate cues. Often, for such complex guidelines, the code underlying the clinical system may also have to be modified to handle such convoluted, multistage protocols if they are “new and unique” in the experience of the CPOE or EHR system. The approach taken will depend on the resources available, the style of the institution, and the capabilities/flexibilities of the clinical system. A host of reference material supporting guideline implementation. With respect to run-time guideline activation, many reluctant clinicians take a “show-me” stance. In such settings, helpful “educational” links explaining both the rationale for the guideline's suggestions (i.e., the “evidence base” for the guideline) as well as detailed explanations of the procedural steps involved in implementing the guideline, including explicit displays of the calculations/logic performed by the program in a patient-specific manner (e.g., dose calculations) are required. Internet and intranet links to national Web sites, locally maintained “expert” monographs, and documents that describe local hospital policy and procedures must support local guideline implementation. Such references require local maintenance above and beyond any “national” guideline content maintenance per se. An organized set of ancillary “EHR system-based” information relevant to clinicians' thought processes to support their attempts to follow a guideline. For example, if the information related to the guideline requires concurrent awareness of active orders, medication doses, laboratory results, etc. (as might occur for a heparin therapy advisor), how will the information be obtained and displayed in “real time” using the underlying CPOE or EHR system as part of the “guideline display page” so that the clinician has “one stop shopping” for guideline-related decision making? There are no pragmatics at the national level for how to do this in individual local systems. A method for advertising the availability/applicability of the guideline for appropriate user groups. This might consist of Placing a guideline order set (if it exists in this form) in the default list of selectable order sets for nursing units on which the guideline is likely to be applicable. Automatically triggering the order set or advisor based on when the user enters specific orders or when specific real-time laboratory results occur. This ability is dependent on the flexibility of the clinical system. A set of parameters and methods for tracking and measuring guideline effectiveness. If the CPOE (local guideline implementation) system does not distinguish in its “log files” (system database) which orders were entered “free hand,” which orders were entered via which “order set,” and which orders were created using a specific “advisor program,” then determining the situations in which guideline suggestions were being followed may become difficult or impossible. Guideline implementers must be able to determine both when a user was prompted to follow a guideline and whether the user chose to do so (and optimally to record why a guideline suggestion was not followed through user-generated explanatory text). The mechanics of doing so are dependent almost wholly on the local system and cannot be specified as part of a nationally distributed “reference guideline document” that is quasi-automatically incorporated into a system (although this might be possible for groups of users using the same clinical system). In addition, because a published guideline is a snapshot in time, local changes in practice and expertise may lead to subsequent customization of some guidelines, so a mechanism for “versioning” both guidelines and their effecter order sets and advisors is required in the local clinical system. The above-described tasks highlight the guideline disseminators' and guideline implementers' mutual problem: Although there is a desire to have a “top down,” document-centric representation that fully describes each guideline, such representations cannot offer pragmatic, easily assimilated, maintainable, and actionable mechanisms for guideline incorporation into local production systems, nor can they do so in a manner that effectively integrates the guideline into local workflows. Pryor and Hripcsak's10 sharing of “rather simple” Arden syntax medical logic modules (MLM) between two institutions a decade ago began to reveal the magnitude of effort required for local integration. In their example, seven MLMs required 43 modifications to be translated between two systems that had already “adopted” the ASTM-standard MLM. A decade later, their conclusion is still applicable: “Standards can be of great assistance in sharing the work of many, but the routine sharing of medical knowledge may be delayed until common standards exist not only in the description of the logic but in all aspects of the medical information system.” As Bates11 suggests, simplicity in implementation often is most effective. Clinical end users sometimes resent overly complex, multiple-screen advisors even when they convey best practices. In implementing clinical systems, it is important to remember that the clinician-user is both more intelligent and more knowledgeable and understanding of the patient's condition than is the clinical computer system. If one relies on the intelligence of the end user (clinician) as a component of guideline implementation and execution, very simple guideline representations, such as order sets, may suffice. Additionally, clinicians should have the ultimate discretion in patient care, including guideline implementation. There are more exceptions than rules in clinical practice. As long as clinicians are made aware of relevant guidelines “just in time” during patient care activities, “optimal” guideline compliance rates may be 80% and not 100% due to patient-specific factors not considered by guideline developers. Effective guideline implementation is hopelessly intertwined with considerations based on local clinical applications, local clinical practices, and local control of procedures and policies as they evolve over time. National document-centric representations of guidelines, although helpful and important, must be seen as supporting the local pragmatics of implementing guidelines on the front lines, and increasing emphasis should be placed on the latter. The situation is not hopeless in that the work that local institutions must do to adopt and maintain guidelines can sometimes be shared “locally” among hospitals and clinics belonging to a conglomerate “system” that share a common information system infrastructure or “locally” among the “user group” of a national vendor with multiple install sites, all of whom presumably share the same implementation platform. The best methods for evolving and supporting guidelines, once they are “installed” will remain an active area of both research and practical interest as EHR and CPOE systems become more ubiquitous.

Most rule learning systems posit hard decision boundaries for continuous attributes and point estimates of rule accuracy, with no measures of variance, which may seem arbitrary to a domain expert. These hard boundaries/points change with small perturbations to the training data. Moreover, rule induction typically produces a large number of rules that must be filtered and interpreted by an analyst. We describe a method of combining rules over multiple bootstrap replications of rule induction so as to reduce the total number of rules presented to an analyst and to provide measures of variance to continuous attribute decision boundaries and accuracy-point estimates. The method is illustrated with perioperative data.