OBJECTIVES: We report on findings from a meeting convened by the American College of Medical Informatics (ACMI) to characterize aspects of the patient experience that could be improved using informatics. MATERIALS AND METHODS: The American College of Medical Informatics fellows were invited to share their experiences as patients and suggest informatics approaches that may improve the patient experience. RESULTS: We identified 4 themes: (1) getting the right care, (2) data sharing and data interoperability, (3) guiding low-cost evaluations, and (4) predictive analytics. DISCUSSION: Despite widespread adoption of health IT, patient experiences remain far from optimal. CONCLUSION: The American College of Medical Informatics fellows identified informatics approaches, applications, and research areas that have the potential to improve patient experiences with health care systems.

OBJECTIVES: To assess the potential to adapt an existing technology regulatory model, namely the Clinical Laboratory Improvement Amendments (CLIA), for clinical artificial intelligence (AI). MATERIALS AND METHODS: We identify overlap in the quality management requirements for laboratory testing and clinical AI. RESULTS: We propose modifications to the CLIA model that could make it suitable for oversight of clinical AI. DISCUSSION: In national discussions of clinical AI, there has been surprisingly little consideration of this longstanding model for local technology oversight. While CLIA was specifically designed for laboratory testing, most of its principles are applicable to other technologies in patient care. CONCLUSION: A CLIA-like approach to regulating clinical AI would be complementary to the more centralized schemes currently under consideration, and it would ensure institutional and professional accountability for the longitudinal quality management of clinical AI.

The Safety Assurance Factors for Electronic Health Record (EHR) Resilience (SAFER) Guides provide recommendations to healthcare organizations for conducting proactive self-assessments of the safety and effectiveness of their EHR implementation and use. Originally released in 2014, they were last updated in 2016. In 2022, the Centers for Medicare and Medicaid Services required their annual attestation by US hospitals. OBJECTIVES: This case study describes how SAFER Guide recommendations were updated to align with current evidence and clinical practice. MATERIALS AND METHODS: Over nine months, a multidisciplinary team updated SAFER Guides through literature reviews, iterative feedback, and online meetings. RESULTS: We reduced the number of recommended practices across all Guides by 40% and consolidated 9 Guides into 8 to maximize ease of use, feasibility, and utility. We provide a 4-level evidence grading hierarchy for each recommendation and a new 5-point rating scale to self-assess implementation status of the recommendation. We included 429 citations of which 289 (67%) were published since the 2016 revision. DISCUSSION: SAFER Guides were revised to offer EHR best practices, adaptable to unique organizational needs, with interactive content available at: https://www.healthit.gov/topic/safety/safer-guides. CONCLUSION: Revisions ensure that the 2025 SAFER Guides represent the best available current evidence for EHR developers and healthcare organizations.

OBJECTIVE: Implement the 5-type health information technology (HIT) patient safety concern classification system for HIT patient safety issues reported to the Veterans Health Administration's Informatics Patient Safety Office. MATERIALS AND METHODS: A team of informatics safety analysts retrospectively classified 1 year of HIT patient safety issues by type of HIT patient safety concern using consensus discussions. The processes established during retrospective classification were then applied to incoming HIT safety issues moving forward. RESULTS: Of 140 issues retrospectively reviewed, 124 met the classification criteria. The majority were HIT failures (eg, software defects) (33.1%) or configuration and implementation problems (29.8%). Unmet user needs and external system interactions accounted for 20.2% and 10.5%, respectively. Absence of HIT safety features accounted for 2.4% of issues, and 4% did not have enough information to classify. CONCLUSION: The 5-type HIT safety concern classification framework generated actionable categories helping organizations effectively respond to HIT patient safety risks.

BACKGROUND: Integrating artificial intelligence (AI) in healthcare settings has the potential to benefit clinical decision-making. Addressing challenges such as ensuring trustworthiness, mitigating bias, and maintaining safety is paramount. The lack of established methodologies for pre- and post-deployment evaluation of AI tools regarding crucial attributes such as transparency, performance monitoring, and adverse event reporting makes this situation challenging. OBJECTIVES: This paper aims to make practical suggestions for creating methods, rules, and guidelines to ensure that the development, testing, supervision, and use of AI in clinical decision support (CDS) systems are done well and safely for patients. MATERIALS AND METHODS: In May 2023, the Division of Clinical Informatics at Beth Israel Deaconess Medical Center and the American Medical Informatics Association co-sponsored a working group on AI in healthcare. In August 2023, there were 4 webinars on AI topics and a 2-day workshop in September 2023 for consensus-building. The event included over 200 industry stakeholders, including clinicians, software developers, academics, ethicists, attorneys, government policy experts, scientists, and patients. The goal was to identify challenges associated with the trusted use of AI-enabled CDS in medical practice. Key issues were identified, and solutions were proposed through qualitative analysis and a 4-month iterative consensus process. RESULTS: Our work culminated in several key recommendations: (1) building safe and trustworthy systems; (2) developing validation, verification, and certification processes for AI-CDS systems; (3) providing a means of safety monitoring and reporting at the national level; and (4) ensuring that appropriate documentation and end-user training are provided. DISCUSSION: AI-enabled Clinical Decision Support (AI-CDS) systems promise to revolutionize healthcare decision-making, necessitating a comprehensive framework for their development, implementation, and regulation that emphasizes trustworthiness, transparency, and safety. This framework encompasses various aspects including model training, explainability, validation, certification, monitoring, and continuous evaluation, while also addressing challenges such as data privacy, fairness, and the need for regulatory oversight to ensure responsible integration of AI into clinical workflow. CONCLUSIONS: Achieving responsible AI-CDS systems requires a collective effort from many healthcare stakeholders. This involves implementing robust safety, monitoring, and transparency measures while fostering innovation. Future steps include testing and piloting proposed trust mechanisms, such as safety reporting protocols, and establishing best practice guidelines.

OBJECTIVE: To develop and evaluate a data-driven process to generate suggestions for improving alert criteria using explainable artificial intelligence (XAI) approaches. METHODS: We extracted data on alerts generated from January 1, 2019 to December 31, 2020, at Vanderbilt University Medical Center. We developed machine learning models to predict user responses to alerts. We applied XAI techniques to generate global explanations and local explanations. We evaluated the generated suggestions by comparing with alert's historical change logs and stakeholder interviews. Suggestions that either matched (or partially matched) changes already made to the alert or were considered clinically correct were classified as helpful. RESULTS: The final dataset included 2 991 823 firings with 2689 features. Among the 5 machine learning models, the LightGBM model achieved the highest Area under the ROC Curve: 0.919 [0.918, 0.920]. We identified 96 helpful suggestions. A total of 278 807 firings (9.3%) could have been eliminated. Some of the suggestions also revealed workflow and education issues. CONCLUSION: We developed a data-driven process to generate suggestions for improving alert criteria using XAI techniques. Our approach could identify improvements regarding clinical decision support (CDS) that might be overlooked or delayed in manual reviews. It also unveils a secondary purpose for the XAI: to improve quality by discovering scenarios where CDS alerts are not accepted due to workflow, education, or staffing issues.

OBJECTIVE: To use workflow execution models to highlight new considerations for patient-centered clinical decision support policies (PC CDS), processes, procedures, technology, and expertise required to support new workflows. METHODS: To generate and refine models, we used (1) targeted literature reviews; (2) key informant interviews with 6 external PC CDS experts; (3) model refinement based on authors' experience; and (4) validation of the models by a 26-member steering committee. RESULTS AND DISCUSSION: We identified 7 major issues that provide significant challenges and opportunities for healthcare systems, researchers, administrators, and health IT and app developers. Overcoming these challenges presents opportunities for new or modified policies, processes, procedures, technology, and expertise to: (1) Ensure patient-generated health data (PGHD), including patient-reported outcomes (PROs), are documented, reviewed, and managed by appropriately trained clinicians, between visits and after regular working hours. (2) Educate patients to use connected medical devices and handle technical issues. (3) Facilitate collection and incorporation of PGHD, PROs, patient preferences, and social determinants of health into existing electronic health records. (4) Troubleshoot erroneous data received from devices. (5) Develop dashboards to display longitudinal patient-reported data. (6) Provide reimbursement to support new models of care. (7) Support patient engagement with remote devices. CONCLUSION: Several new policies, processes, technologies, and expertise are required to ensure safe and effective implementation and use of PC CDS. As we gain more experience implementing and working with PC CDS, we should be able to begin realizing the long-term positive impact on patient health that the patient-centered movement in healthcare promises.

OBJECTIVE: To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. METHODS: We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. RESULTS: Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. CONCLUSION: AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.

OBJECTIVE: Measures of diagnostic performance in cancer are underdeveloped. Electronic clinical quality measures (eCQMs) to assess quality of cancer diagnosis could help quantify and improve diagnostic performance. MATERIALS AND METHODS: We developed 2 eCQMs to assess diagnostic evaluation of red-flag clinical findings for colorectal (CRC; based on abnormal stool-based cancer screening tests or labs suggestive of iron deficiency anemia) and lung (abnormal chest imaging) cancer. The 2 eCQMs quantified rates of red-flag follow-up in CRC and lung cancer using electronic health record data repositories at 2 large healthcare systems. Each measure used clinical data to identify abnormal results, evidence of appropriate follow-up, and exclusions that signified follow-up was unnecessary. Clinicians reviewed 100 positive and 20 negative randomly selected records for each eCQM at each site to validate accuracy and categorized missed opportunities related to system, provider, or patient factors. RESULTS: We implemented the CRC eCQM at both sites, while the lung cancer eCQM was only implemented at the VA due to lack of structured data indicating level of cancer suspicion on most chest imaging results at Geisinger. For the CRC eCQM, the rate of appropriate follow-up was 36.0% (26 746/74 314 patients) in the VA after removing clinical exclusions and 41.1% at Geisinger (1009/2461 patients; P < .001). Similarly, the rate of appropriate evaluation for lung cancer in the VA was 61.5% (25 166/40 924 patients). Reviewers most frequently attributed missed opportunities at both sites to provider factors (84 of 157). CONCLUSIONS: We implemented 2 eCQMs to evaluate the diagnostic process in cancer at 2 large health systems. Health care organizations can use these eCQMs to monitor diagnostic performance related to cancer.

The design, development, implementation, use, and evaluation of high-quality, patient-centered clinical decision support (PC CDS) is necessary if we are to achieve the quintuple aim in healthcare. We developed a PC CDS lifecycle framework to promote a common understanding and language for communication among researchers, patients, clinicians, and policymakers. The framework puts the patient, and/or their caregiver at the center and illustrates how they are involved in all the following stages: Computable Clinical Knowledge, Patient-specific Inference, Information Delivery, Clinical Decision, Patient Behaviors, Health Outcomes, Aggregate Data, and patient-centered outcomes research (PCOR) Evidence. Using this idealized framework reminds key stakeholders that developing, deploying, and evaluating PC-CDS is a complex, sociotechnical challenge that requires consideration of all 8 stages. In addition, we need to ensure that patients, their caregivers, and the clinicians caring for them are explicitly involved at each stage to help us achieve the quintuple aim.

OBJECTIVE: To improve problem list documentation and care quality. MATERIALS AND METHODS: We developed algorithms to infer clinical problems a patient has that are not recorded on the coded problem list using structured data in the electronic health record (EHR) for 12 clinically significant heart, lung, and blood diseases. We also developed a clinical decision support (CDS) intervention which suggests adding missing problems to the problem list. We evaluated the intervention at 4 diverse healthcare systems using 3 different EHRs in a randomized trial using 3 predetermined outcome measures: alert acceptance, problem addition, and National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) clinical quality measures. RESULTS: There were 288 832 opportunities to add a problem in the intervention arm and the problem was added 63 777 times (acceptance rate 22.1%). The intervention arm had 4.6 times as many problems added as the control arm. There were no significant differences in any of the clinical quality measures. DISCUSSION: The CDS intervention was highly effective at improving problem list completeness. However, the improvement in problem list utilization was not associated with improvement in the quality measures. The lack of effect on quality measures suggests that problem list documentation is not directly associated with improvements in quality measured by National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) quality measures. However, improved problem list accuracy has other benefits, including clinical care, patient comprehension of health conditions, accurate CDS and population health, and for research. CONCLUSION: An EHR-embedded CDS intervention was effective at improving problem list completeness but was not associated with improvement in quality measures.

Supporting healthcare decision-making that is patient-centered and evidence-based requires investments in the development of tools and techniques for dissemination of patient-centered outcomes research findings via methods such as clinical decision support (CDS). This article explores the technical landscape for patient-centered CDS (PC CDS) and the gaps in making PC CDS more shareable, standards-based, and publicly available, with the goal of improving patient care and clinical outcomes. This landscape assessment used: (1) a technical expert panel; (2) a literature review; and (3) interviews with 18 CDS stakeholders. We identified 7 salient technical considerations that span 5 phases of PC CDS development. While progress has been made in the technical landscape, the field must advance standards for translating clinical guidelines into PC CDS, the standardization of CDS insertion points into the clinical workflow, and processes to capture, standardize, and integrate patient-generated health data.

OBJECTIVE: We conducted a horizon scan to (1) identify challenges in patient-centered clinical decision support (PC CDS) and (2) identify future directions for PC CDS. MATERIALS AND METHODS: We engaged a technical expert panel, conducted a scoping literature review, and interviewed key informants. We qualitatively analyzed literature and interview transcripts, mapping findings to the 4 phases for translating evidence into PC CDS interventions (Prioritizing, Authoring, Implementing, and Measuring) and to external factors. RESULTS: We identified 12 challenges for PC CDS development. Lack of patient input was identified as a critical challenge. The key informants noted that patient input is critical to prioritizing topics for PC CDS and to ensuring that CDS aligns with patients' routine behaviors. Lack of patient-centered terminology standards was viewed as a challenge in authoring PC CDS. We found a dearth of CDS studies that measured clinical outcomes, creating significant gaps in our understanding of PC CDS' impact. Across all phases of CDS development, there is a lack of patient and provider trust and limited attention to patients' and providers' concerns. DISCUSSION: These challenges suggest opportunities for advancing PC CDS. There are opportunities to develop industry-wide practices and standards to increase transparency, standardize terminologies, and incorporate patient input. There is also opportunity to engage patients throughout the PC CDS research process to ensure that outcome measures are relevant to their needs. CONCLUSION: Addressing these challenges and embracing these opportunities will help realize the promise of PC CDS-placing patients at the center of the healthcare system.

How to deliver best care in various clinical settings remains a vexing problem. All pertinent healthcare-related questions have not, cannot, and will not be addressable with costly time- and resource-consuming controlled clinical trials. At present, evidence-based guidelines can address only a small fraction of the types of care that clinicians deliver. Furthermore, underserved areas rarely can access state-of-the-art evidence-based guidelines in real-time, and often lack the wherewithal to implement advanced guidelines. Care providers in such settings frequently do not have sufficient training to undertake advanced guideline implementation. Nevertheless, in advanced modern healthcare delivery environments, use of eActions (validated clinical decision support systems) could help overcome the cognitive limitations of overburdened clinicians. Widespread use of eActions will require surmounting current healthcare technical and cultural barriers and installing clinical evidence/data curation systems. The authors expect that increased numbers of evidence-based guidelines will result from future comparative effectiveness clinical research carried out during routine healthcare delivery within learning healthcare systems.

Over the next decade, many health care organizations (HCOs) will transition from one electronic health record (EHR) to another; some forced by hospital acquisition and others by choice in search of better EHRs. Herein, we apply principles of Requisite Imagination, or the ability to imagine key aspects of the future one is planning, to offer 6 recommendations on how to proactively safeguard these transitions. First, HCOs should implement a proactive leadership structure that values communication. Second, HCOs should implement proactive risk assessment and testing processes. Third, HCOs should anticipate and reduce unwarranted variation in their EHR and clinical processes. Fourth, HCOs should establish a culture of conscious inquiry with routine system monitoring. Fifth, HCOs should foresee and reduce information access problems. Sixth, HCOs should support their workforce through difficult EHR transitions. Proactive approaches using Requisite Imagination principles outlined here can help ensure safe, effective, and economically sound EHR transitions.

Addressing environmental pollution and climate change is one of the biggest sociotechnical challenges of our time. While information technology has led to improvements in healthcare, it has also contributed to increased energy usage, destructive natural resource extraction, piles of e-waste, and increased greenhouse gases. We introduce a framework "Information technology-enabled Clinical cLimate InforMAtics acTions for the Environment" (i-CLIMATE) to illustrate how clinical informatics can help reduce healthcare's environmental pollution and climate-related impacts using 5 actionable components: (1) create a circular economy for health IT, (2) reduce energy consumption through smarter use of health IT, (3) support more environmentally friendly decision-making by clinicians and health administrators, (4) mobilize healthcare workforce environmental stewardship through informatics, and (5) Inform policies and regulations for change. We define Clinical Climate Informatics as a field that applies data, information, and knowledge management principles to operationalize components of the i-CLIMATE Framework.

OBJECTIVE: To identify common medication route-related causes of clinical decision support (CDS) malfunctions and best practices for avoiding them. MATERIALS AND METHODS: Case series of medication route-related CDS malfunctions from diverse healthcare provider organizations. RESULTS: Nine cases were identified and described, including both false-positive and false-negative alert scenarios. A common cause was the inclusion of nonsystemically available medication routes in value sets (eg, eye drops, ear drops, or topical preparations) when only systemically available routes were appropriate. DISCUSSION: These value set errors are common, occur across healthcare provider organizations and electronic health record (EHR) systems, affect many different types of medications, and can impact the accuracy of CDS interventions. New knowledge management tools and processes for auditing existing value sets and supporting the creation of new value sets can mitigate many of these issues. Furthermore, value set issues can adversely affect other aspects of the EHR, such as quality reporting and population health management. CONCLUSION: Value set issues related to medication routes are widespread and can lead to CDS malfunctions. Organizations should make appropriate investments in knowledge management tools and strategies, such as those outlined in our recommendations.

OBJECTIVE: Accurate and robust quality measurement is critical to the future of value-based care. Having incomplete information when calculating quality measures can cause inaccuracies in reported patient outcomes. This research examines how quality calculations vary when using data from an individual electronic health record (EHR) and longitudinal data from a health information exchange (HIE) operating as a multisource registry for quality measurement. MATERIALS AND METHODS: Data were sampled from 53 healthcare organizations in 2018. Organizations represented both ambulatory care practices and health systems participating in the state of Kansas HIE. Fourteen ambulatory quality measures for 5300 patients were calculated using the data from an individual EHR source and contrasted to calculations when HIE data were added to locally recorded data. RESULTS: A total of 79% of patients received care at more than 1 facility during the 2018 calendar year. A total of 12 994 applicable quality measure calculations were compared using data from the originating organization vs longitudinal data from the HIE. A total of 15% of all quality measure calculations changed (P < .001) when including HIE data sources, affecting 19% of patients. Changes in quality measure calculations were observed across measures and organizations. DISCUSSION: These results demonstrate that quality measures calculated using single-site EHR data may be limited by incomplete information. Effective data sharing significantly changes quality calculations, which affect healthcare payments, patient safety, and care quality. CONCLUSIONS: Federal, state, and commercial programs that use quality measurement as part of reimbursement could promote more accurate and representative quality measurement through methods that increase clinical data sharing.

Clinical decision-making is based on knowledge, expertise, and authority, with clinicians approving almost every intervention-the starting point for delivery of "All the right care, but only the right care," an unachieved healthcare quality improvement goal. Unaided clinicians suffer from human cognitive limitations and biases when decisions are based only on their training, expertise, and experience. Electronic health records (EHRs) could improve healthcare with robust decision-support tools that reduce unwarranted variation of clinician decisions and actions. Current EHRs, focused on results review, documentation, and accounting, are awkward, time-consuming, and contribute to clinician stress and burnout. Decision-support tools could reduce clinician burden and enable replicable clinician decisions and actions that personalize patient care. Most current clinical decision-support tools or aids lack detail and neither reduce burden nor enable replicable actions. Clinicians must provide subjective interpretation and missing logic, thus introducing personal biases and mindless, unwarranted, variation from evidence-based practice. Replicability occurs when different clinicians, with the same patient information and context, come to the same decision and action. We propose a feasible subset of therapeutic decision-support tools based on credible clinical outcome evidence: computer protocols leading to replicable clinician actions (eActions). eActions enable different clinicians to make consistent decisions and actions when faced with the same patient input data. eActions embrace good everyday decision-making informed by evidence, experience, EHR data, and individual patient status. eActions can reduce unwarranted variation, increase quality of clinical care and research, reduce EHR noise, and could enable a learning healthcare system.

The development and implementation of clinical decision support (CDS) that trains itself and adapts its algorithms based on new data-here referred to as Adaptive CDS-present unique challenges and considerations. Although Adaptive CDS represents an expected progression from earlier work, the activities needed to appropriately manage and support the establishment and evolution of Adaptive CDS require new, coordinated initiatives and oversight that do not currently exist. In this AMIA position paper, the authors describe current and emerging challenges to the safe use of Adaptive CDS and lay out recommendations for the effective management and monitoring of Adaptive CDS.

OBJECTIVE: Diagnostic errors are major contributors to preventable patient harm. We validated the use of an electronic health record (EHR)-based trigger (e-trigger) to measure missed opportunities in stroke diagnosis in emergency departments (EDs). METHODS: Using two frameworks, the Safer Dx Trigger Tools Framework and the Symptom-disease Pair Analysis of Diagnostic Error Framework, we applied a symptom-disease pair-based e-trigger to identify patients hospitalized for stroke who, in the preceding 30 days, were discharged from the ED with benign headache or dizziness diagnoses. The algorithm was applied to Veteran Affairs National Corporate Data Warehouse on patients seen between 1/1/2016 and 12/31/2017. Trained reviewers evaluated medical records for presence/absence of missed opportunities in stroke diagnosis and stroke-related red-flags, risk factors, neurological examination, and clinical interventions. Reviewers also estimated quality of clinical documentation at the index ED visit. RESULTS: We applied the e-trigger to 7,752,326 unique patients and identified 46,931 stroke-related admissions, of which 398 records were flagged as trigger-positive and reviewed. Of these, 124 had missed opportunities (positive predictive value for "missed" = 31.2%), 93 (23.4%) had no missed opportunity (non-missed), 162 (40.7%) were miscoded, and 19 (4.7%) were inconclusive. Reviewer agreement was high (87.3%, Cohen's kappa = 0.81). Compared to the non-missed group, the missed group had more stroke risk factors (mean 3.2 vs 2.6), red flags (mean 0.5 vs 0.2), and a higher rate of inadequate documentation (66.9% vs 28.0%). CONCLUSION: In a large national EHR repository, a symptom-disease pair-based e-trigger identified missed diagnoses of stroke with a modest positive predictive value, underscoring the need for chart review validation procedures to identify diagnostic errors in large data sets.

OBJECTIVE: The study sought to determine availability and use of structured override reasons for drug-drug interaction (DDI) alerts in electronic health records. MATERIALS AND METHODS: We collected data on DDI alerts and override reasons from 10 clinical sites across the United States using a variety of electronic health records. We used a multistage iterative card sort method to categorize the override reasons from all sites and identified best practices. RESULTS: Our methodology established 177 unique override reasons across the 10 sites. The number of coded override reasons at each site ranged from 3 to 100. Many sites offered override reasons not relevant to DDIs. Twelve categories of override reasons were identified. Three categories accounted for 78% of all overrides: "will monitor or take precautions," "not clinically significant," and "benefit outweighs risk." DISCUSSION: We found wide variability in override reasons between sites and many opportunities to improve alerts. Some override reasons were irrelevant to DDIs. Many override reasons attested to a future action (eg, decreasing a dose or ordering monitoring tests), which requires an additional step after the alert is overridden, unless the alert is made actionable. Some override reasons deferred to another party, although override reasons often are not visible to other users. Many override reasons stated that the alert was inaccurate, suggesting that specificity of alerts could be improved. CONCLUSIONS: Organizations should improve the options available to providers who choose to override DDI alerts. DDI alerting systems should be actionable and alerts should be tailored to the patient and drug pairs.

Objective: Online portals provide patients with access to their test results, but it is unknown how patients use these tools to manage results and what information is available to promote understanding. We conducted a mixed-methods study to explore patients' experiences and preferences when accessing their test results via portals. Materials and Methods: We conducted 95 interviews (13 semistructured and 82 structured) with adults who viewed a test result in their portal between April 2015 and September 2016 at 4 large outpatient clinics in Houston, Texas. Semistructured interviews were coded using content analysis and transformed into quantitative data and integrated with the structured interview data. Descriptive statistics were used to summarize the structured data. Results: Nearly two-thirds (63%) did not receive any explanatory information or test result interpretation at the time they received the result, and 46% conducted online searches for further information about their result. Patients who received an abnormal result were more likely to experience negative emotions (56% vs 21%; P = .003) and more likely to call their physician (44% vs 15%; P = .002) compared with those who received normal results. Discussion: Study findings suggest that online portals are not currently designed to present test results to patients in a meaningful way. Patients experienced negative emotions often with abnormal results, but sometimes even with normal results. Simply providing access via portals is insufficient; additional strategies are needed to help patients interpret and manage their online test results. Conclusion: Given the absence of national guidance, our findings could help strengthen policy and practice in this area and inform innovations that promote patient understanding of test results.

Objective: We assessed changes in the percentage of providers with positive perceptions of electronic health record (EHR) benefit before and after transition from a local basic to a commercial comprehensive EHR. Methods: Changes in the percentage of providers with positive perceptions of EHR benefit were captured via a survey of academic health care providers before (baseline) and at 6-12 months (short term) and 12-24 months (long term) after the transition. We analyzed 32 items for the overall group and by practice setting, provider age, and specialty using separate multivariable-adjusted random effects logistic regression models. Results: A total of 223 providers completed all 3 surveys (30% response rate): 85.6% had outpatient practices, 56.5% were >45 years old, and 23.8% were primary care providers. The percentage of providers with positive perceptions significantly increased from baseline to long-term follow-up for patient communication, hospital transitions - access to clinical information, preventive care delivery, preventive care prompt, preventive lab prompt, satisfaction with system reliability, and sharing medical information (P < .05 for each). The percentage of providers with positive perceptions significantly decreased over time for overall satisfaction, productivity, better patient care, clinical decision quality, easy access to patient information, monitoring patients, more time for patients, coordination of care, computer access, adequate resources, and satisfaction with ease of use (P < 0.05 for each). Results varied by subgroup. Conclusion: After a transition to a commercial comprehensive EHR, items with significant increases and significant decreases in the percentage of providers with positive perceptions of EHR benefit were identified, overall and by subgroup.

Objective: To assess the impact of electronic health record (EHR) implementation on hospital finances. Materials and Methods: We analyzed the impact of EHR implementation on bond ratings and net income from service to patients (NISP) at 32 hospitals that recently implemented a new EHR and a set of controls. Results: After implementing an EHR, 7 hospitals had a bond downgrade, 7 had a bond upgrade, and 18 had no changes. There was no difference in the likelihood of bond rating changes or in changes to NISP following EHR go-live when compared to control hospitals. Discussion: Most hospitals in our analysis saw no change in bond ratings following EHR go-live, with no significant differences observed between EHR implementation and control hospitals. There was also no apparent difference in NISP. Conclusions: Implementation of an EHR did not appear to have an impact on bond ratings at the hospitals in our analysis.

Objective: The Safety Assurance Factors for EHR Resilience (SAFER) guides were released in 2014 to help health systems conduct proactive risk assessment of electronic health record (EHR)- safety related policies, processes, procedures, and configurations. The extent to which SAFER recommendations are followed is unknown. Methods: We conducted risk assessments of 8 organizations of varying size, complexity, EHR, and EHR adoption maturity. Each organization self-assessed adherence to all 140 unique SAFER recommendations contained within 9 guides (range 10-29 recommendations per guide). In each guide, recommendations were organized into 3 broad domains: "safe health IT" (total 45 recommendations); "using health IT safely" (total 80 recommendations); and "monitoring health IT" (total 15 recommendations). Results: The 8 sites fully implemented 25 of 140 (18%) SAFER recommendations. Mean number of "fully implemented" recommendations per guide ranged from 94% (System Interfaces-18 recommendations) to 63% (Clinical Communication-12 recommendations). Adherence was higher for "safe health IT" domain (82.1%) vs "using health IT safely" (72.5%) and "monitoring health IT" (67.3%). Conclusions: Despite availability of recommendations on how to improve use of EHRs, most recommendations were not fully implemented. New national policy initiatives are needed to stimulate implementation of these best practices.

Objective: To identify and understand the factors that contribute to medication errors associated with the use of computerized provider order entry (CPOE) in pediatrics and provide recommendations on how CPOE systems could be improved. Materials and Methods: We conducted a systematic literature review across 3 large databases: the Cumulative Index to Nursing and Allied Health Literature, Embase, and Medline. Three independent reviewers screened the titles, and 2 authors then independently reviewed all abstracts and full texts, with 1 author acting as a constant across all publications. Data were extracted onto a customized data extraction sheet, and a narrative synthesis of all eligible studies was undertaken. Results: A total of 47 articles were included in this review. We identified 5 factors that contributed to errors with the use of a CPOE system: (1) lack of drug dosing alerts, which failed to detect calculation errors; (2) generation of inappropriate dosing alerts, such as warnings based on incorrect drug indications; (3) inappropriate drug duplication alerts, as a result of the system failing to consider factors such as the route of administration; (4) dropdown menu selection errors; and (5) system design issues, such as a lack of suitable dosing options for a particular drug. Discussion and Conclusions: This review highlights 5 key factors that contributed to the occurrence of CPOE-related medication errors in pediatrics. Dosing support is the most important. More advanced clinical decision support that can suggest doses based on the drug indication is needed.

Objective: To develop an empirically derived taxonomy of clinical decision support (CDS) alert malfunctions. Materials and Methods: We identified CDS alert malfunctions using a mix of qualitative and quantitative methods: (1) site visits with interviews of chief medical informatics officers, CDS developers, clinical leaders, and CDS end users; (2) surveys of chief medical informatics officers; (3) analysis of CDS firing rates; and (4) analysis of CDS overrides. We used a multi-round, manual, iterative card sort to develop a multi-axial, empirically derived taxonomy of CDS malfunctions. Results: We analyzed 68 CDS alert malfunction cases from 14 sites across the United States with diverse electronic health record systems. Four primary axes emerged: the cause of the malfunction, its mode of discovery, when it began, and how it affected rule firing. Build errors, conceptualization errors, and the introduction of new concepts or terms were the most frequent causes. User reports were the predominant mode of discovery. Many malfunctions within our database caused rules to fire for patients for whom they should not have (false positives), but the reverse (false negatives) was also common. Discussion: Across organizations and electronic health record systems, similar malfunction patterns recurred. Challenges included updates to code sets and values, software issues at the time of system upgrades, difficulties with migration of CDS content between computing environments, and the challenge of correctly conceptualizing and building CDS. Conclusion: CDS alert malfunctions are frequent. The empirically derived taxonomy formalizes the common recurring issues that cause these malfunctions, helping CDS developers anticipate and prevent CDS malfunctions before they occur or detect and resolve them expediently.

Clinical vocabularies allow for standard representation of clinical concepts, and can also contain knowledge structures, such as hierarchy, that facilitate the creation of maintainable and accurate clinical decision support (CDS). A key architectural feature of clinical hierarchies is how they handle parent-child relationships - specifically whether hierarchies are strict hierarchies (allowing a single parent per concept) or polyhierarchies (allowing multiple parents per concept). These structures handle subsumption relationships (ie, ancestor and descendant relationships) differently. In this paper, we describe three real-world malfunctions of clinical decision support related to incorrect assumptions about subsumption checking for β-blocker, specifically carvedilol, a non-selective β-blocker that also has α-blocker activity. We recommend that 1) CDS implementers should learn about the limitations of terminologies, hierarchies, and classification, 2) CDS implementers should thoroughly test CDS, with a focus on special or unusual cases, 3) CDS implementers should monitor feedback from users, and 4) electronic health record (EHR) and clinical content developers should offer and support polyhierarchical clinical terminologies, especially for medications.

A clinical decision support system and its components may malfunction due to different reasons. The objective of this work is to develop computational methods that can help us to monitor the system and assure its proper operation by promptly detecting and analyzing changes in its behavior. We develop a new change-point detection method using the Multi-Process Dynamic Linear Model. The experiments on real and simulated data show that our method outperforms existing change-point detection methods, leading to higher accuracy and shorter delay in the detection.

OBJECTIVE: To understand the different types and causes of prescribing errors associated with computerized provider order entry (CPOE) systems, and recommend improvements in these systems. MATERIALS AND METHODS: We conducted a systematic review of the literature published between January 2004 and June 2015 using three large databases: the Cumulative Index to Nursing and Allied Health Literature, Embase, and Medline. Studies that reported qualitative data about the types and causes of these errors were included. A narrative synthesis of all eligible studies was undertaken. RESULTS: A total of 1185 publications were identified, of which 34 were included in the review. We identified 8 key themes associated with CPOE-related prescribing errors: computer screen display, drop-down menus and auto-population, wording, default settings, nonintuitive or inflexible ordering, repeat prescriptions and automated processes, users' work processes, and clinical decision support systems. Displaying an incomplete list of a patient's medications on the computer screen often contributed to prescribing errors. Lack of system flexibility resulted in users employing error-prone workarounds, such as the addition of contradictory free-text comments. Users' misinterpretations of how text was presented in CPOE systems were also linked with the occurrence of prescribing errors. DISCUSSION AND CONCLUSIONS: Human factors design is important to reduce error rates. Drop-down menus should be designed with safeguards to decrease the likelihood of selection errors. Development of more sophisticated clinical decision support, which can perform checks on free-text, may also prevent errors. Further research is needed to ensure that systems minimize error likelihood and meet users' workflow expectations.

Objective: The United States Office of the National Coordinator for Health Information Technology sponsored the development of a "high-priority" list of drug-drug interactions (DDIs) to be used for clinical decision support. We assessed current adoption of this list and current alerting practice for these DDIs with regard to alert implementation (presence or absence of an alert) and display (alert appearance as interruptive or passive). Materials and methods: We conducted evaluations of electronic health records (EHRs) at a convenience sample of health care organizations across the United States using a standardized testing protocol with simulated orders. Results: Evaluations of 19 systems were conducted at 13 sites using 14 different EHRs. Across systems, 69% of the high-priority DDI pairs produced alerts. Implementation and display of the DDI alerts tested varied between systems, even when the same EHR vendor was used. Across the drug pairs evaluated, implementation and display of DDI alerts differed, ranging from 27% (4/15) to 93% (14/15) implementation. Discussion: Currently, there is no standard of care covering which DDI alerts to implement or how to display them to providers. Opportunities to improve DDI alerting include using differential displays based on DDI severity, establishing improved lists of clinically significant DDIs, and thoroughly reviewing organizational implementation decisions regarding DDIs. Conclusion: DDI alerting is clinically important but not standardized. There is significant room for improvement and standardization around evidence-based DDIs.

OBJECTIVE: Methods to identify and study safety risks of electronic health records (EHRs) are underdeveloped and largely depend on limited end-user reports. "Safety huddles" have been found useful in creating a sense of collective situational awareness that increases an organization's capacity to respond to safety concerns. We explored the use of safety huddles for identifying and learning about EHR-related safety concerns. DESIGN: Data were obtained from daily safety huddle briefing notes recorded at a single midsized tertiary-care hospital in the United States over 1 year. Huddles were attended by key administrative, clinical, and information technology staff. We conducted a content analysis of huddle notes to identify what EHR-related safety concerns were discussed. We expanded a previously developed EHR-related error taxonomy to categorize types of EHR-related safety concerns recorded in the notes. RESULTS: On review of daily huddle notes spanning 249 days, we identified 245 EHR-related safety concerns. For our analysis, we defined EHR technology to include a specific EHR functionality, an entire clinical software application, or the hardware system. Most concerns (41.6%) involved " EHR technology working incorrectly, " followed by 25.7% involving " EHR technology not working at all. " Concerns related to "EHR technology missing or absent" accounted for 16.7%, whereas 15.9% were linked to " user errors ." CONCLUSIONS: Safety huddles promoted discussion of several technology-related issues at the organization level and can serve as a promising technique to identify and address EHR-related safety concerns. Based on our findings, we recommend that health care organizations consider huddles as a strategy to promote understanding and improvement of EHR safety.

In this report, we describe 2 instances in which expert use of an electronic health record (EHR) system interfaced to an external clinical laboratory information system led to unintended consequences wherein 2 patients failed to have laboratory tests drawn in a timely manner. In both events, user actions combined with the lack of an acknowledgment message describing the order cancellation from the external clinical system were the root causes. In 1 case, rapid, near-simultaneous order entry was the culprit; in the second, astute order management by a clinician, unaware of the lack of proper 2-way interface messaging from the external clinical system, led to the confusion. Although testing had shown that the laboratory system would cancel duplicate laboratory orders, it was thought that duplicate alerting in the new order entry system would prevent such events.

Thorough and ongoing testing of electronic health records (EHRs) is key to ensuring their safety and effectiveness. Many health care organizations limit testing to test environments separate from, and often different than, the production environment used by clinicians. Because EHRs are complex hardware and software systems that often interact with other hardware and software systems, no test environment can exactly mimic how the production environment will behave. An effective testing process must integrate safely conducted testing in the production environment itself, using test patients. We propose recommendations for how to safely incorporate testing in production into current EHR testing practices, with suggestions regarding the incremental release of upgrades, test patients, tester accounts, downstream personnel, and reporting.

AMIA is leading the effort to strengthen the health informatics profession by creating an advanced health informatics certification (AHIC) for individuals whose informatics work directly impacts the practice of health care, public health, or personal health. The AMIA Board of Directors has endorsed a set of proposed AHIC eligibility requirements that will be presented to the future AHIC certifying entity for adoption. These requirements specifically establish who will be eligible to sit for the AHIC examination and more generally signal the depth and breadth of knowledge and experience expected from certified individuals. They also inform the development of the accreditation process and provide guidance to graduate health informatics programs as well as individuals interested in pursuing AHIC. AHIC eligibility will be determined by practice focus, education in primary field and health informatics, and significant health informatics experience.

OBJECTIVE: To illustrate ways in which clinical decision support systems (CDSSs) malfunction and identify patterns of such malfunctions. MATERIALS AND METHODS: We identified and investigated several CDSS malfunctions at Brigham and Women's Hospital and present them as a case series. We also conducted a preliminary survey of Chief Medical Information Officers to assess the frequency of such malfunctions. RESULTS: We identified four CDSS malfunctions at Brigham and Women's Hospital: (1) an alert for monitoring thyroid function in patients receiving amiodarone stopped working when an internal identifier for amiodarone was changed in another system; (2) an alert for lead screening for children stopped working when the rule was inadvertently edited; (3) a software upgrade of the electronic health record software caused numerous spurious alerts to fire; and (4) a malfunction in an external drug classification system caused an alert to inappropriately suggest antiplatelet drugs, such as aspirin, for patients already taking one. We found that 93% of the Chief Medical Information Officers who responded to our survey had experienced at least one CDSS malfunction, and two-thirds experienced malfunctions at least annually. DISCUSSION: CDSS malfunctions are widespread and often persist for long periods. The failure of alerts to fire is particularly difficult to detect. A range of causes, including changes in codes and fields, software upgrades, inadvertent disabling or editing of rules, and malfunctions of external systems commonly contribute to CDSS malfunctions, and current approaches for preventing and detecting such malfunctions are inadequate. CONCLUSION: CDSS malfunctions occur commonly and often go undetected. Better methods are needed to prevent and detect these malfunctions.

OBJECTIVE: Clinical decision support (CDS) is essential for delivery of high-quality, cost-effective, and safe healthcare. The authors sought to evaluate the CDS capabilities across electronic health record (EHR) systems. METHODS: We evaluated the CDS implementation capabilities of 8 Office of the National Coordinator for Health Information Technology Authorized Certification Body (ONC-ACB)-certified EHRs. Within each EHR, the authors attempted to implement 3 user-defined rules that utilized the various data and logic elements expected of typical EHRs and that represented clinically important evidenced-based care. The rules were: 1) if a patient has amiodarone on his or her active medication list and does not have a thyroid-stimulating hormone (TSH) result recorded in the last 12 months, suggest ordering a TSH; 2) if a patient has a hemoglobin A1c result >7% and does not have diabetes on his or her problem list, suggest adding diabetes to the problem list; and 3) if a patient has coronary artery disease on his or her problem list and does not have aspirin on the active medication list, suggest ordering aspirin. RESULTS: Most evaluated EHRs lacked some CDS capabilities; 5 EHRs were able to implement all 3 rules, and the remaining 3 EHRs were unable to implement any of the rules. One of these did not allow users to customize CDS rules at all. The most frequently found shortcomings included the inability to use laboratory test results in rules, limit rules by time, use advanced Boolean logic, perform actions from the alert interface, and adequately test rules. CONCLUSION: Significant improvements in the EHR certification and implementation procedures are necessary.

The Office of the National Coordinator for Health Information Technology is expected to oversee creation of a Health Information Technology (HIT) Safety Center. While its functions are still being defined, the center is envisioned as a public-private entity focusing on promotion of HIT related patient safety. We propose that the HIT Safety Center leverages its unique position to work with key administrative and policy stakeholders, healthcare organizations (HCOs), and HIT vendors to achieve four goals: (1) facilitate creation of a nationwide 'post-marketing' surveillance system to monitor HIT related safety events; (2) develop methods and governance structures to support investigation of major HIT related safety events; (3) create the infrastructure and methods needed to carry out random assessments of HIT related safety in complex HCOs; and (4) advocate for HIT safety with government and private entities. The convening ability of a federally supported HIT Safety Center could be critically important to our transformation to a safe and effective HIT enabled healthcare system.

Accurate display and interpretation of clinical laboratory test results is essential for safe and effective diagnosis and treatment. In an attempt to ascertain how well current electronic health records (EHRs) facilitated these processes, we evaluated the graphical displays of laboratory test results in eight EHRs using objective criteria for optimal graphs based on literature and expert opinion. None of the EHRs met all 11 criteria; the magnitude of deficiency ranged from one EHR meeting 10 of 11 criteria to three EHRs meeting only 5 of 11 criteria. One criterion (i.e., the EHR has a graph with y-axis labels that display both the name of the measured variable and the units of measure) was absent from all EHRs. One EHR system graphed results in reverse chronological order. One EHR system plotted data collected at unequally-spaced points in time using equally-spaced data points, which had the effect of erroneously depicting the visual slope perception between data points. This deficiency could have a significant, negative impact on patient safety. Only two EHR systems allowed users to see, hover-over, or click on a data point to see the precise values of the x-y coordinates. Our study suggests that many current EHR-generated graphs do not meet evidence-based criteria aimed at improving laboratory data comprehension.

We have identified 5 use cases that comprise a useful definition of an "open or interoperable electronic health record (EHR)." Each of these use cases represents important functionality that should be available to 1) clinicians, so they can provide safe and effective health care; 2) researchers, so they can advance our understanding of disease and health care processes; 3) administrators, so they can reduce their reliance on a single-source EHR developer; 4) software developers, so they can develop innovative solutions to address limitations of current EHR user interfaces and new applications to improve the practice of medicine; and 5) patients, so they can access their personal health information no matter where they receive their health care. Widespread access to "open EHRs" that can accommodate at least these 5 use cases is important if we are to realize the enormous potential of EHR-enabled health care systems.

OBJECTIVES: We conducted a systematic review to determine the effect of providing patients access to their medical records (electronic or paper-based) on healthcare quality, as defined by measures of safety, effectiveness, patient-centeredness, timeliness, efficiency, and equity. METHODS: Articles indexed in PubMed from January 1970 to January 2012 were reviewed. Twenty-seven English-language controlled studies were included. Outcomes were categorized as measures of effectiveness (n=19), patient-centeredness (n=16), and efficiency (n=2); no study addressed safety, timeliness, or equity. RESULTS: Outcomes were equivocal with respect to several aspects of effectiveness and patient-centeredness. Efficiency outcomes in terms of frequency of in-person and telephone encounters were mixed. Access to health records appeared to enhance patients' perceptions of control and reduced or had no effect on patient anxiety. CONCLUSION: Although few positive findings generally favored patient access, the literature is unclear on whether providing patients access to their medical records improves quality.

INTRODUCTION: Clinical databases require accurate entity resolution (ER). One approach is to use algorithms that assign questionable cases to manual review. Few studies have compared the performance of common algorithms for such a task. Furthermore, previous work has been limited by a lack of objective methods for setting algorithm parameters. We compared the performance of common ER algorithms: using algorithmic optimization, rather than manual parameter tuning, and on two-threshold classification (match/manual review/non-match) as well as single-threshold (match/non-match). METHODS: We manually reviewed 20,000 randomly selected, potential duplicate record-pairs to identify matches (10,000 training set, 10,000 test set). We evaluated the probabilistic expectation maximization, simple deterministic and fuzzy inference engine (FIE) algorithms. We used particle swarm to optimize algorithm parameters for a single and for two thresholds. We ran 10 iterations of optimization using the training set and report averaged performance against the test set. RESULTS: The overall estimated duplicate rate was 6%. FIE and simple deterministic algorithms allowed a lower manual review set compared to the probabilistic method (FIE 1.9%, simple deterministic 2.5%, probabilistic 3.6%; p<0.001). For a single threshold, the simple deterministic algorithm performed better than the probabilistic method (positive predictive value 0.956 vs 0.887, sensitivity 0.985 vs 0.887, p<0.001). ER with FIE classifies 98.1% of record-pairs correctly (1/10,000 error rate), assigning the remainder to manual review. CONCLUSIONS: Optimized deterministic algorithms outperform the probabilistic method. There is a strong case for considering optimized deterministic methods for ER.

OBJECTIVE: Electronic health records (EHR) hold great promise for managing patient information in ways that improve healthcare delivery. Physicians differ, however, in their use of this health information technology (IT), and these differences are not well understood. The authors study the differences in individual physicians' EHR use patterns and identify perceptions of uncertainty as an important new variable in understanding EHR use. DESIGN: Qualitative study using semi-structured interviews and direct observation of physicians (n=28) working in a multispecialty outpatient care organization. MEASUREMENTS: We identified physicians' perceptions of uncertainty as an important variable in understanding differences in EHR use patterns. Drawing on theories from the medical and organizational literatures, we identified three categories of perceptions of uncertainty: reduction, absorption, and hybrid. We used an existing model of EHR use to categorize physician EHR use patterns as high, medium, and low based on degree of feature use, level of EHR-enabled communication, and frequency that EHR use patterns change. RESULTS: Physicians' perceptions of uncertainty were distinctly associated with their EHR use patterns. Uncertainty reductionists tended to exhibit high levels of EHR use, uncertainty absorbers tended to exhibit low levels of EHR use, and physicians demonstrating both perspectives of uncertainty (hybrids) tended to exhibit medium levels of EHR use. CONCLUSIONS: We find evidence linking physicians' perceptions of uncertainty with EHR use patterns. Study findings have implications for health IT research, practice, and policy, particularly in terms of impacting health IT design and implementation efforts in ways that consider differences in physicians' perceptions of uncertainty.

OBJECTIVE: A recent Institute of Medicine report called for attention to safety issues related to electronic health records (EHRs). We analyzed EHR-related safety concerns reported within a large, integrated healthcare system. METHODS: The Informatics Patient Safety Office of the Veterans Health Administration (VA) maintains a non-punitive, voluntary reporting system to collect and investigate EHR-related safety concerns (ie, adverse events, potential events, and near misses). We analyzed completed investigations using an eight-dimension sociotechnical conceptual model that accounted for both technical and non-technical dimensions of safety. Using the framework analysis approach to qualitative data, we identified emergent and recurring safety concerns common to multiple reports. RESULTS: We extracted 100 consecutive, unique, closed investigations between August 2009 and May 2013 from 344 reported incidents. Seventy-four involved unsafe technology and 25 involved unsafe use of technology. A majority (70%) involved two or more model dimensions. Most often, non-technical dimensions such as workflow, policies, and personnel interacted in a complex fashion with technical dimensions such as software/hardware, content, and user interface to produce safety concerns. Most (94%) safety concerns related to either unmet data-display needs in the EHR (ie, displayed information available to the end user failed to reduce uncertainty or led to increased potential for patient harm), software upgrades or modifications, data transmission between components of the EHR, or 'hidden dependencies' within the EHR. DISCUSSION: EHR-related safety concerns involving both unsafe technology and unsafe use of technology persist long after 'go-live' and despite the sophisticated EHR infrastructure represented in our data source. Currently, few healthcare institutions have reporting and analysis capabilities similar to the VA. CONCLUSIONS: Because EHR-related safety concerns have complex sociotechnical origins, institutions with long-standing as well as recent EHR implementations should build a robust infrastructure to monitor and learn from them.

OBJECTIVE: To describe the activities performed by people involved in clinical decision support (CDS) at leading sites. MATERIALS AND METHODS: We conducted ethnographic observations at seven diverse sites with a history of excellence in CDS using the Rapid Assessment Process and analyzed the data using a series of card sorts, informed by Linstone's Multiple Perspectives Model. RESULTS: We identified 18 activities and grouped them into four areas. Area 1: Fostering relationships across the organization, with activities (a) training and support, (b) visibility/presence on the floor, (c) liaising between people, (d) administration and leadership, (e) project management, (f) cheerleading/buy-in/sponsorship, (g) preparing for CDS implementation. Area 2: Assembling the system with activities (a) providing technical support, (b) CDS content development, (c) purchasing products from vendors (d) knowledge management, (e) system integration. Area 3: Using CDS to achieve the organization's goals with activities (a) reporting, (b) requirements-gathering/specifications, (c) monitoring CDS, (d) linking CDS to goals, (e) managing data. Area 4: Participation in external policy and standards activities (this area consists of only a single activity). We also identified a set of recommendations associated with these 18 activities. DISCUSSION: All 18 activities we identified were performed at all sites, although the way they were organized into roles differed substantially. We consider these activities critical to the success of a CDS program. CONCLUSIONS: A series of activities are performed by sites strong in CDS, and sites adopting CDS should ensure they incorporate these activities into their efforts.

Larry Weed, MD is widely known as the father of the problem-oriented medical record and inventor of the now-ubiquitous SOAP (subjective/objective/assessment/plan) note, for developing an electronic health record system (Problem-Oriented Medical Information System, PROMIS), and for founding a company (since acquired), which developed problem-knowledge couplers. However, Dr Weed's vision for medicine goes far beyond software--over the course of his storied career, he has relentlessly sought to bring the scientific method to medical practice and, where necessary, to point out shortcomings in the system and advocate for change. In this oral history, Dr Weed describes, in his own words, the arcs of his long career and the work that remains to be done.

CONTEXT: Failure to notify patients of test results is common even when electronic health records (EHRs) are used to report results to practitioners. We sought to understand the broad range of social and technical factors that affect test result management in an integrated EHR-based health system. METHODS: Between June and November 2010, we conducted a cross-sectional, web-based survey of all primary care practitioners (PCPs) within the Department of Veterans Affairs nationwide. Survey development was guided by a socio-technical model describing multiple inter-related dimensions of EHR use. FINDINGS: Of 5001 PCPs invited, 2590 (51.8%) responded. 55.5% believed that the EHRs did not have convenient features for notifying patients of test results. Over a third (37.9%) reported having staff support needed for notifying patients of test results. Many relied on the patient's next visit to notify them for normal (46.1%) and abnormal results (20.1%). Only 45.7% reported receiving adequate training on using the EHR notification system and 35.1% reported having an assigned contact for technical assistance with the EHR; most received help from colleagues (60.4%). A majority (85.6%) stayed after hours or came in on weekends to address notifications; less than a third reported receiving protected time (30.1%). PCPs strongly endorsed several new features to improve test result management, including better tracking and visualization of result notifications. CONCLUSIONS: Despite an advanced EHR, both social and technical challenges exist in ensuring notification of test results to practitioners and patients. Current EHR technology requires significant improvement in order to avoid similar challenges elsewhere.

BACKGROUND: Electronic health record (EHR) users must regularly review large amounts of data in order to make informed clinical decisions, and such review is time-consuming and often overwhelming. Technologies like automated summarization tools, EHR search engines and natural language processing have been shown to help clinicians manage this information. OBJECTIVE: To develop a support vector machine (SVM)-based system for identifying EHR progress notes pertaining to diabetes, and to validate it at two institutions. MATERIALS AND METHODS: We retrieved 2000 EHR progress notes from patients with diabetes at the Brigham and Women's Hospital (1000 for training and 1000 for testing) and another 1000 notes from the University of Texas Physicians (for validation). We manually annotated all notes and trained a SVM using a bag of words approach. We then used the SVM on the testing and validation sets and evaluated its performance with the area under the curve (AUC) and F statistics. RESULTS: The model accurately identified diabetes-related notes in both the Brigham and Women's Hospital testing set (AUC=0.956, F=0.934) and the external University of Texas Faculty Physicians validation set (AUC=0.947, F=0.935). DISCUSSION: Overall, the model we developed was quite accurate. Furthermore, it generalized, without loss of accuracy, to another institution with a different EHR and a distinct patient and provider population. CONCLUSIONS: It is possible to use a SVM-based classifier to identify EHR progress notes pertaining to diabetes, and the model generalizes well.

OBJECTIVE: Computerized provider order entry (CPOE) with clinical decision support (CDS) can help hospitals improve care. Little is known about what CDS is presently in use and how it is managed, however, especially in community hospitals. This study sought to address this knowledge gap by identifying standard practices related to CDS in US community hospitals with mature CPOE systems. MATERIALS AND METHODS: Representatives of 34 community hospitals, each of which had over 5 years experience with CPOE, were interviewed to identify standard practices related to CDS. Data were analyzed with a mix of descriptive statistics and qualitative approaches to the identification of patterns, themes and trends. RESULTS: This broad sample of community hospitals had robust levels of CDS despite their small size and the independent nature of many of their physician staff members. The hospitals uniformly used medication alerts and order sets, had sophisticated governance procedures for CDS, and employed staff to customize CDS. DISCUSSION: The level of customization needed for most CDS before implementation was greater than expected. Customization requires skilled individuals who represent an emerging manpower need at this type of hospital. CONCLUSION: These results bode well for robust diffusion of CDS to similar hospitals in the process of adopting CDS and suggest that national policies to promote CDS use may be successful.

OBJECTIVE: We describe a novel, crowdsourcing method for generating a knowledge base of problem-medication pairs that takes advantage of manually asserted links between medications and problems. METHODS: Through iterative review, we developed metrics to estimate the appropriateness of manually entered problem-medication links for inclusion in a knowledge base that can be used to infer previously unasserted links between problems and medications. RESULTS: Clinicians manually linked 231,223 medications (55.30% of prescribed medications) to problems within the electronic health record, generating 41,203 distinct problem-medication pairs, although not all were accurate. We developed methods to evaluate the accuracy of the pairs, and after limiting the pairs to those meeting an estimated 95% appropriateness threshold, 11,166 pairs remained. The pairs in the knowledge base accounted for 183,127 total links asserted (76.47% of all links). Retrospective application of the knowledge base linked 68,316 medications not previously linked by a clinician to an indicated problem (36.53% of unlinked medications). Expert review of the combined knowledge base, including inferred and manually linked problem-medication pairs, found a sensitivity of 65.8% and a specificity of 97.9%. CONCLUSION: Crowdsourcing is an effective, inexpensive method for generating a knowledge base of problem-medication pairs that is automatically mapped to local terminologies, up-to-date, and reflective of local prescribing practices and trends.

Using an eight-dimensional model for studying socio-technical systems, a multidisciplinary team of investigators identified barriers and facilitators to clinical decision support (CDS) implementation in a community setting, the Mid-Valley Independent Physicians Association in the Salem, Oregon area. The team used the Rapid Assessment Process, which included nine formal interviews with CDS stakeholders, and observation of 27 clinicians. The research team, which has studied 21 healthcare sites of various sizes over the past 12 years, believes this site is an excellent example of an organization which is using a commercially available electronic-health-record system with CDS well. The eight-dimensional model proved useful as an organizing structure for the evaluation.

OBJECTIVE: Clinical decision support (CDS) is a powerful tool for improving healthcare quality and ensuring patient safety; however, effective implementation of CDS requires effective clinical and technical governance structures. The authors sought to determine the range and variety of these governance structures and identify a set of recommended practices through observational study. DESIGN: Three site visits were conducted at institutions across the USA to learn about CDS capabilities and processes from clinical, technical, and organizational perspectives. Based on the results of these visits, written questionnaires were sent to the three institutions visited and two additional sites. Together, these five organizations encompass a variety of academic and community hospitals as well as small and large ambulatory practices. These organizations use both commercially available and internally developed clinical information systems. MEASUREMENTS: Characteristics of clinical information systems and CDS systems used at each site as well as governance structures and content management approaches were identified through extensive field interviews and follow-up surveys. RESULTS: Six recommended practices were identified in the area of governance, and four were identified in the area of content management. Key similarities and differences between the organizations studied were also highlighted. CONCLUSION: Each of the five sites studied contributed to the recommended practices presented in this paper for CDS governance. Since these strategies appear to be useful at a diverse range of institutions, they should be considered by any future implementers of decision support.

BACKGROUND: Clinical decision support (CDS) is a valuable tool for improving healthcare quality and lowering costs. However, there is no comprehensive taxonomy of types of CDS and there has been limited research on the availability of various CDS tools across current electronic health record (EHR) systems. OBJECTIVE: To develop and validate a taxonomy of front-end CDS tools and to assess support for these tools in major commercial and internally developed EHRs. STUDY DESIGN AND METHODS: We used a modified Delphi approach with a panel of 11 decision support experts to develop a taxonomy of 53 front-end CDS tools. Based on this taxonomy, a survey on CDS tools was sent to a purposive sample of commercial EHR vendors (n=9) and leading healthcare institutions with internally developed state-of-the-art EHRs (n=4). RESULTS: Responses were received from all healthcare institutions and 7 of 9 EHR vendors (response rate: 85%). All 53 types of CDS tools identified in the taxonomy were found in at least one surveyed EHR system, but only 8 functions were present in all EHRs. Medication dosing support and order facilitators were the most commonly available classes of decision support, while expert systems (eg, diagnostic decision support, ventilator management suggestions) were the least common. CONCLUSION: We developed and validated a comprehensive taxonomy of front-end CDS tools. A subsequent survey of commercial EHR vendors and leading healthcare institutions revealed a small core set of common CDS tools, but identified significant variability in the remainder of clinical decision support content.

OBJECTIVE: Electronic medical records (EMRs) facilitate abnormal test result communication through "alert" notifications. The aim was to evaluate how primary care providers (PCPs) manage alerts related to critical diagnostic test results on their EMR screens, and compare alert-management strategies of providers with high versus low rates of timely follow-up of results. DESIGN: 28 PCPs from a large, tertiary care Veterans Affairs Medical Center (VAMC) were purposively sampled according to their rates of timely follow-up of alerts, determined in a previous study. Using techniques from cognitive task analysis, participants were interviewed about how and when they manage alerts, focusing on four alert-management features to filter, sort and reduce unnecessary alerts on their EMR screens. RESULTS: Provider knowledge of alert-management features ranged between 4% and 75%. Almost half (46%) of providers did not use any of these features, and none used more than two. Providers with higher versus lower rates of timely follow-up used the four features similarly, except one (customizing alert notifications). Providers with low rates of timely follow-up tended to manually scan the alert list and process alerts heuristically using their clinical judgment. Additionally, 46% of providers used at least one workaround strategy to manage alerts. CONCLUSION: Considerable heterogeneity exists in provider use of alert-management strategies; specific strategies may be associated with lower rates of timely follow-up. Standardization of alert-management strategies including improving provider knowledge of appropriate tools in the EMR to manage alerts could reduce the lack of timely follow-up of abnormal diagnostic test results.

BACKGROUND: The most effective decision support systems are integrated with clinical information systems, such as inpatient and outpatient electronic health records (EHRs) and computerized provider order entry (CPOE) systems. Purpose The goal of this project was to describe and quantify the results of a study of decision support capabilities in Certification Commission for Health Information Technology (CCHIT) certified electronic health record systems. METHODS: The authors conducted a series of interviews with representatives of nine commercially available clinical information systems, evaluating their capabilities against 42 different clinical decision support features. RESULTS: Six of the nine reviewed systems offered all the applicable event-driven, action-oriented, real-time clinical decision support triggers required for initiating clinical decision support interventions. Five of the nine systems could access all the patient-specific data items identified as necessary. Six of the nine systems supported all the intervention types identified as necessary to allow clinical information systems to tailor their interventions based on the severity of the clinical situation and the user's workflow. Only one system supported all the offered choices identified as key to allowing physicians to take action directly from within the alert. Discussion The principal finding relates to system-by-system variability. The best system in our analysis had only a single missing feature (from 42 total) while the worst had eighteen.This dramatic variability in CDS capability among commercially available systems was unexpected and is a cause for concern. CONCLUSIONS: These findings have implications for four distinct constituencies: purchasers of clinical information systems, developers of clinical decision support, vendors of clinical information systems and certification bodies.

BACKGROUND: Computerized provider order entry (CPOE) systems can help hospitals improve health care quality, but they can also introduce new problems. The extent to which hospitals experience unintended consequences of CPOE, which include more than errors, has not been quantified in prior research. OBJECTIVE: To discover the extent and importance of unintended adverse consequences related to CPOE implementation in U.S. hospitals. DESIGN, SETTING, AND PARTICIPANTS: Building on a prior qualitative study involving fieldwork at five hospitals, we developed and then administered a telephone survey concerning the extent and importance of CPOE-related unintended adverse consequences to representatives from 176 hospitals in the U.S. that have CPOE. MEASUREMENTS: Self report by key informants of the extent and level of importance to the overall function of the hospital of eight types of unintended adverse consequences experienced by sites with inpatient CPOE. RESULTS We found that hospitals experienced all eight types of unintended adverse consequences, although respondents identified several they considered more important than others. Those related to new work/more work, workflow, system demands, communication, emotions, and dependence on the technology were ranked as most severe, with at least 72% of respondents ranking them as moderately to very important. Hospital representatives are less sure about shifts in the power structure and CPOE as a new source of errors. There is no relation between kinds of unintended consequences and number of years CPOE has been used. Despite the relatively short length of time most hospitals have had CPOE (median five years), it is highly infused, or embedded, within work practice at most of these sites. CONCLUSIONS: The unintended consequences of CPOE are widespread and important to those knowledgeable about CPOE in hospitals. They can be positive, negative, or both, depending on one's perspective, and they continue to exist over the duration of use. Aggressive detection and management of adverse unintended consequences is vital for CPOE success.

We agree with Weiner et al. that adoption of the term “e-Iatrogenesis”1 is both timely and necessary as we begin to identify what we have called “new kinds of errors”2 resulting from CPOE implementation. The computerization of clinical information capture with software tools such as CPOE will doubtlessly cause the emergence of a wide variety of new kinds of errors: “e-Iatrogenesis” provides a clear and concise rubric for these unintended and undesired consequences. In mid-December, 2006, the London Times reported on the inadvertent prescription of Viagra to a set of patients in the United Kingom.3 This error occurred when general practitioners using the UK's National Health Service “e-Formulary” attempted to prescribe “Zyban” (a medication commonly used to assist patients in smoking cessation). The system mistakenly selected “sildenafil” (the generic name for Viagra) instead. A spokesperson for the NHS denied that any untoward effects had resulted, and that immediate steps had been taken to rectify the error, including notifying over 900 practitioners at more than 300 clinics warning them of the potential problem for their patients.

Personal health records (PHRs) hold great promise for empowering patients and increasing the accuracy and completeness of health information. We reviewed two small USB-based PHR devices that allow a patient to easily store and transport their personal health information. Both devices offer password protection and encryption features. Analysis of the devices shows that they store their data in a Microsoft Access database. Due to a flaw in the encryption of this database, recovering the user's password can be accomplished with minimal effort. Our analysis also showed that, rather than encrypting health information with the password chosen by the user, the devices stored the user's password as a string in the database and then encrypted that database with a common password set by the manufacturer. This is another serious vulnerability. This article describes the weaknesses we discovered, outlines three critical flaws with the security model used by the devices, and recommends four guidelines for improving the security of similar devices.

OBJECTIVE: To identify types of clinical unintended adverse consequences resulting from computerized provider order entry (CPOE) implementation. DESIGN: An expert panel provided initial examples of adverse unintended consequences of CPOE. The authors, using qualitative methods, gathered and analyzed additional examples from five successful CPOE sites. METHODS: Using a card sort method, the authors developed a categorization scheme for the 79 unintended consequences initially identified and then iteratively modified the scheme to categorize 245 additional adverse consequences resulting from fieldwork. Because the focus centered on consequences requiring prevention or remedial action, the authors did not further analyze reported unintended beneficial (positive) consequences. RESULTS: Unintended adverse consequences (UACs) fell into nine major categories (in order of decreasing frequency): 1) more/new work for clinicians; 2) unfavorable workflow issues; 3) never ending system demands; 4) problems related to paper persistence; 5) untoward changes in communication patterns and practices; 6) negative emotions; 7) generation of new kinds of errors; 8) unexpected changes in the power structure; and 9) overdependence on the technology. Clinical decision support features introduced many of these unintended consequences. CONCLUSION: Identifying and understanding the types and in some instances the causes of unintended adverse consequences associated with CPOE will enable system developers and implementers to better manage implementation and maintenance of future CPOE projects.

“The army called me down to New York [in 1950]. I was with New York University (NYU)—and the colonel said to me, ‘Well, if you volunteer to be in the army, then you'll become a lieutenant, an officer. But if you don't volunteer, you'll be drafted anyway, and sent to boot camp. So I volunteered. And they sent me to medical field service school in Fort Sam Houston, Texas. And that was kind of interesting. And then, I guess my card dropped out, they wanted a dentist who was a physicist. And that was me.” Robert S. Ledley Dr. Robert S. Ledley is credited with “sowing the seeds” for the field of medical informatics [Ledley 1959],1 initiating the development of computerized medical image analysis [Ledley 1964],2 and for being the principal investigator of the Protein Information Resource (PIR) for 20 years [Dayhoff 1965].3 Dr. Ledley is best known for developing the first whole-body computerized tomography (CT or CAT) scanner in 1973 (Patent No. 3,922,552), which revolutionized diagnostic medicine. Dr. Ledley's first CT scanner [which he called the Automatic Computerized Transverse Axial, or ACTA, scanner; [Ledley 1974a, 1974b]2, 4 is now owned by the Smithsonian Institution's Museum of American History [Kondratas 2005].5 Using his scanner, he was the first to perform three-dimensional reconstructions [Huang 1975],6 the first to use CT in radiation therapy planning for cancer patients [Scheer 1977],7 and the creator of many other “firsts” in the application of CT in medicine. The following story describes his efforts to develop this scanner. I always liked physics. I started at Columbia [University in 1942]. Once you’re in the college, you can take any course in the whole university. In those days they only gave one year of college physics, and everything else was graduate physics. At Columbia, [graduate physics] was incredible. I had something like eight Nobel Prize winners as teachers. Hans A. Bethe,a for example, came down from Ithaca and gave a course on atomic physics. It was absolutely incredible. And I had [Enrico] Fermib in thermodynamics. Yeah, it was a fantastic place. I received my masters' degree in [Theoretical] Physics in 1949. I went to dental school because my father said, “You know, a physicist can't make a living, really. And the only thing you can do is be a poor professor. What you should do is become a dentist”—and we had a number of dentists in our family—“and then you can support your love of physics with your dentistry.” And it turns out that I knew something about the history of physics. In the late 1800s and early 1900s, that's what was done. Most people supported their physics by doing other things.c The only one who I knew who didn't do this was [Isidor Isaac] Rabi,d and he was a professor and head of the physics department at Columbia. And he thought the idea that I was going to dental school at the same time I was taking graduate courses in physics was hysterical. It was the funniest thing he had ever known about. I graduated from Dental School in 1948, but I was too young to get a license. You had to be twenty-one to get a license. I moved to Georgetown University in 1970. I moved the [National Biomedical Research] Foundation and everything, right in here, into these quarters [lower level of the medical library at Georgetown University]. And I had big grants from NIH. I had an engineering equipment grant, to build equipment that was required. And I also had a programming grant to do all the basic programming work that was required. And somebody said to me, “Why don't you put the two grants together, and it’d be a unified grant.” Fine, okay, so I put them together—NIH, whatever they say, you do. And then Nixon was president, and I get a call from NIH that the grants were approved, ready to be paid. This was in May. They were going to be paid in June. [Then] I get a call from NIH, “Nixon cut back the medical research funds”—“and we don't have the money to pay you.” I had about, I don't know, forty, fifty people working. So I had to figure something quick! So I talked to John Rose, hopefully my savior, and he said, “Wow. Well, you do scanning, right?” [RL:] “Yeah.” He said, “Well, Dr. Luessenhop is interested in buying a scanner.” So I went to see Dr. Luessenhop. Nice fellow, chief of Neurosurgery, but he was a brain surgeon. He says, “Well, I was going to buy this machine,” and he shows me a slick brochure, the EMI machine [Hounsfield, 1976].8 It was a head scanner only. But what he didn't know is that they didn't have it yet. They just had a brochure! So I looked at this book, and sure enough, you could see organs, but they were fuzzy. It was terrible; picture after picture. I didn't see how they could use much of that. But at the same time, I read a paper in The Journal of Applied Physics. It was written by a fellow by the name of Cormack at Tufts [Cormack 1963].9 And he took a cylinder of aluminum and a cylinder of wood, with a hole in it, that the aluminum fit into, and he scanned it, one scan. And he said, “Well, it's circular symmetric, so I’ll make a whole lot of scans, because they're all the same.” Okay? And then he worked out the absorption coefficient of the aluminum and the wood, to a very large number of significant figures. And I thought to myself, “Man! that's great! We can do it!” And I figured that I could make it. So I went back to Dr. Luessenhop and I said, “I can make it, and it’ll be half the price” [of that quoted for the EMI scanner]; I didn't even know what the price was! It was a miserable machine. But I knew what to do. I was going to use convolution.e How did I know that? Well, Science magazine began putting pictures on their covers in the late ’60s, I think. And they had a picture of a virus. They made the picture of the cross-section of a virus, and it looked kind of interesting. They took an electron microscope, and they scanned the virus right across. And then they couldn't really rotate it, so they said it's circularly symmetric, and they did what Cormack had done. Anyway, what they did use was [convolution]. Great! People were talking about [convolutions] because you could take scratches out of pictures and things like that. So I figured, “That's it!” And I made a deal with the university. $250,000 of the foundation's money, and we’ll make Dr. Luessenhop a machine that can scan every place on the whole body. A better one! Well, I said I did. (laughter) If you haven't done it, you don't know if you can do it. So we went ahead. I had it designed in my mind, and I drew it on a paper, and we had a lot of cooperation. There was a machine shop—you can't believe it—in Georgetown, on Wisconsin Avenue, just below M Street. Can you imagine? A machine shop run by Allan Mitchell. And I went down, and this guy had a big lathe. Okay, great, so that's the guy. So I went to him, because he had a big lathe. I told him, “I'm making a medical machine.” “Oh, you’re making a medical machine? Oh, that's wonderful. I'd love to help you out.” Of course, they charged, but still. And he said, “I’ve got one fellow, a young man I just hired, and he's very smart, very clever guy, and I’ll put him on it, totally, he will be your machinist! He’ll do nothing else.” Because I knew it had a lot of parts. Then I went around to get a mechanical engineer—couldn't get a mechanical engineer. I went to every engineering school in the region, and I couldn't get an engineer to do this for me. I could show the machinist what I wanted. So I figured, “Well, I’ve already spent two weeks on this, going around. Forget it, I’ l do it myself.” After all, I had had a course in engineering mechanics. So I designed every detail in it myself. There isn't a thing that I learned that I didn't use. So I sat down and designed it in detail. I also had a great electronic engineer, Tom Golab, and a great programmer/mathematician, James Wilson. And what I did was, I drew a sort of three-dimensional picture to show what each part was, and then I made the ordinary mechanical engineering drawings [See Fig. 1 for a copy of the mechanical drawing included in the original patent application]. And this fellow, Frank Rabbitt was his name, and he was great. He had all the sense that was required. So I drew him the pictures, and we’d leave them under the door at about eleven in the evening. It was an around-the-clock job. And then in the morning he’d study the plans when he came in at six in the morning, and he would call me up and say, “Do you really mean you want the hole here? And which hole is it going to be attached to? Are you sure?” And I looked at it, sure enough, he was right. You’ve got a figure that's off slightly he’d say. He was a great guy, and I hired him afterwards. But he never had any engineering training, but he had the engineering sense. Mechanical drawing of the ACTA scanner from Dr. Ledley's patent. So I decided to use cold-rolled steel; that was the strongest material for its weight [at that time]. So I found out about that, and I [went to] take a look at a piece of cold-rolled steel. It looked terrible! It was blotchy, looked like it was rusty, but it wasn't rusty. And I decided the doctors would never go along with that. It's got to look pretty. So Capital Cadillac in those days was down here. So I figured, well, car places have paint shops. So I go down to Capital Cadillac and I say, “I'm making this medical machine.” “Oh! You’re making a medical machine? We’d love to help you out.” So I said, “Cold-rolled steel looks terrible, and I need it painted before it comes here, right from the machine shop.” He said, “I have the ideal person. I’ll put him totally on your job.” And it was a man, who was ready to retire, but he still had time to go, another six months or so, and he said, “This fellow is a detail man. He’ll do the greatest paint job. I hate to give him big jobs, because he can't carry things. He's an older guy.” So that's what I did. So I designed the parts at Georgetown University, I went to Allan Mitchell, they machined it. And the fellow took it over to Capital Cadillac. He was supposed to call me when he got it, which he did, and then he painted it, and then we brought it back. I don't remember whether he brought it, or we went over there to get it. And all I had to do was ream out the threads, because he painted everything. So that's how we made it. That's what we did. And it was a resounding success.f [See Figs. 2 and 3.] The 0100 ACTA Scanner (which stands for automated, computerized, transverse axial scanner) is the world's first whole-body scanner. The computer used to run the ACTA Scanner. Oh! it feels so great. I gave a lecture the other day at the hospital over in Virginia—Fairfax Hospital. I gave this lecture on something about the history, this kind of thing, telling these stories. And a lady stands up and says to me, “You know, I'm walking because of you. I had a bad disease, and it was diagnosed by one of your machines. I'm a nurse now. I am always on my feet, and I always have to walk around. And if they didn't make the diagnosis, I wouldn't be walking today.” So that's the reward, when you really get down to it. I’ll tell you the story of my first life-saving. We were fixing up the machine and making it a little better, this, that, and the other, and Dr. Luessenhop comes in the room, and he looks at the pictures that I made, there were no CT phantoms. You know what a phantom is—a model that you make. They were models that generally were exactly like what you were going to take X-rays of—to see whether something worked—that was the purpose of the model. I figured the hardest thing was going to be the brain. And what I used was the skull, a real skull that I had when I went to school. I took a course in gross anatomy, had to have a real skull. Nowadays, you don't have to have a real skull, but you did then. And in those days there were butchers, and I had delivered to the university, every day, fresh calf's brains. Calf's brains were a delicacy in those days. What were they called—[sweetbreads]? And then I put the calf's brains in the skull, and put test tubes of water in for the ventricles, and scanned them. That was my Phantom. So he said, “That's good enough. I'm bringing down a patient.” This couldn't be done today [because of the Institutional Review Board procedures]! So we scanned a patient, and that was the beginning. Now, after about two weeks, there was a pediatric neurosurgeon, Dr. McCulloch. I got to know all the neurosurgeons. And he says to me on a Monday morning, “Did you hear about the wonderful case we had?” So I said, “Well, I'm sure I heard about it, because I must have seen it, because I haven't walked away from that computer.” I was there eight hours a day with that machine. I couldn't get myself to walk away. It was built into the room in which it was going to be used, and I couldn't get myself to walk away from it. I just couldn't walk away. So he said, “Well, we had a four-year-old boy who fell off a bike and hurt his head. Nobody saw him fall, but the parents took him to a pediatrician. He was kind of groggy, and the pediatrician brought him into the emergency room.” This is what Dr. McCulloch is telling. And he says, “So I happened to be in the emergency room at the time. So I looked at the kid, and I thought to myself, ‘You know, I think we ought to scan him.’ So I took him upstairs and I scanned him.” So I said, “Well, who ran the machine? Because I didn't see any four-year-old boy. There wasn't any four-year-old boy. What are you telling me about?” I stood there every day, eight hours a day, I was standing there, all the time the machine was being used. So I said, “Who ran the machine?” He said, “I did.” I said, “You did?! How’d you know what to do?” And he says, “I watched the tech do it. I’ve seen it.” I said, “And you just knew how to use the machine?” He says, “Yeah! And there was a bleed. And I took him down to Surgery and took out the blood, and you saved his life.” It was incredible! He said, “If I'd waited till Monday, then it would have been over, the kid would have been dead.” That was it. Can you imagine that? That was our first save. And I thought to myself, “What am I standing around for? This neurosurgeon just watched and he could run the machine. I don't have to watch!” [This happened] over and over again, and I was inducted into the National Inventors Hall of Fame. And they had a banquet at the museum. There were ten people at each big round table. After dinner, this lady comes over to me, and she gives me a big bear hug. I mean, she was big herself, she was tall. So I said, “Now, what did I do to deserve that?” And she says, “You saved my daughter's life.” So that's what makes it worthwhile. Well, I started my own company to make the scanners. I was manufacturing the scanners. Well, it was almost that bad. The vice-president in charge of the medical center, a fellow by the name of Matt McNulty—he passed away—but he comes down one day and says, “A lot of people want this thing, don't they?” [He meant] radiologists. I said, “Oh, yeah!” He says, “Well, you really can't go into business in the basement here.” I said, “Yeah, I know.” So he says, “Why don't you just form a profit-making company and bill them?” I said, “Okay, I’ll do that.” So I did, I formed a company called DISCO, Digital Information Science Corporation.g And when Disco was about four or five years old, it was difficult to fill all the orders. Not only that, but there was no venture capital in those days, no new venture capital. And somebody says to me, “Well, you go to the bank and you borrow money.” Okay, so I go to the bank, and I tell the guy I want to borrow money. And I’ve already told him all the things I'm doing. And he says to me, “Could you tell me again exactly what you’re doing?” So I tell him again, and he says, “Well, we actually only who build So I figured of that, I'm my time. So I said to the when they wanted it course the only paid me So I had to build with in And then when I'm half another [Then] when I it, the That's the I did it. That was the the time we got the to the time we it was about I hired all the That's the Now, what happened after a of I decided I’ve got to this I mean, this is too of the was, the in That the out of me, because I absolutely had to build that machine for gave me or And that's what I'm going to do. any about it, I’ve got to People that give me for it, they're going to get their they're going to get it. And that me up at It was a big We had a or And then I had a I delivered the parts to the room in which it was to be Well, first of all, they had to their And then I sent around an to make sure that they the It had to have water to the an to water They had to have the right and and everything. There was a lot to do. What I didn't was that these people made this a of of their I didn't that I went to one the first It was in in I said, “Do you have your room And he said, “Yeah, I have it all and he me to the “You want to see I walk and it had It was at Georgetown, they had the that you ever It was So that was a real good machine [Ledley Dr. Ledley was in in He a degree in physics and from Columbia University and a from the New York University of in In to the ACTA scanner, Dr. Ledley the first purpose image called the the for used by the and many and one of the first machines. He the first for on computer and in used for He was inducted into the National Inventors Hall of in for his of the first CT scanner, and is a of the of for the National of In he was the National of by He was a fellow of the American of and was the in by the Digital and New and New American of and Digital New of in and New New and A. of Dental and National Biomedical and for Computerized and and of the in and Computerized and and of from first whole-body computerized and for first purpose image Digital and In Dr. Ledley or over

MediClass is a knowledge-based system that processes both free-text and coded data to automatically detect clinical events in electronic medical records (EMRs). This technology aims to optimize both clinical practice and process control by automatically coding EMR contents regardless of data input method (e.g., dictation, structured templates, typed narrative). We report on the design goals, implemented functionality, generalizability, and current status of the system. MediClass could aid both clinical operations and health services research through enhancing care quality assessment, disease surveillance, and adverse event detection.

OBJECTIVES: Computer-based provider order entry (CPOE) systems are implemented to increase both efficiency and accuracy in health care, but these systems often cause a myriad of emotions to arise. This qualitative research investigates the emotions surrounding CPOE implementation and use. METHODS: We performed a secondary analysis of several previously collected qualitative data sets from interviews and observations of over 50 individuals. Three researchers worked in parallel to identify themes that expressed emotional responses to CPOE. We then reviewed and classified these quotes using a validated hierarchical taxonomy of semantically homogeneous terms associated with specific emotions. RESULTS: The implementation and use of CPOE systems provoked examples of positive, negative, and neutral emotions. Negative emotional responses were the most prevalent, by far, in all the observations. CONCLUSION: Designing and implementing CPOE systems is difficult. These systems and the implementation process itself often inspire intense emotions. If designers and implementers fail to recognize that various CPOE features and implementation strategies can increase clinicians' negative emotions, then the systems may fail to become a routine part of the clinical care delivery process. We might alleviate some of these problems by designing positive feedback mechanisms for both the systems and the organizations.

Clinical decision support (CDS) in electronic prescribing (eRx) systems can improve the safety, quality, efficiency, and cost-effectiveness of care. However, at present, these potential benefits have not been fully realized. In this consensus white paper, we set forth recommendations and action plans in three critical domains: (1) advances in system capabilities, including basic and advanced sets of CDS interventions and knowledge, supporting database elements, operational features to improve usability and measure performance, and management and governance structures; (2) uniform standards, vocabularies, and centralized knowledge structures and services that could reduce rework by vendors and care providers, improve dissemination of well-constructed CDS interventions, promote generally applicable research in CDS methods, and accelerate the movement of new medical knowledge from research to practice; and (3) appropriate financial and legal incentives to promote adoption.

Computer technology from the management of individual patient medical records to the tracking of epidemiologic trends has become an essential part of all aspects of modern medicine. Consequently, computers, including bedside components, point-of-care testing equipment, and handheld computer devices, are increasingly present in patients' rooms. Recent articles have indicated that computer hardware, just as other medical equipment, may act as a reservoir for microorganisms and contribute to the transfer of pathogens to patients. This article presents basic microbiological concepts relative to infection, reviews the present literature concerning possible links between computer contamination and nosocomial colonizations and infections, discusses basic principles for the control of contamination, and provides guidelines for reducing the risk of transfer of microorganisms to susceptible patient populations.

A Usenet newsgroup, sci.med.informatics, has been created to serve as an international electronic forum for discussion of issues related to medical informatics. The creation process follows a set of administrative rules set out by the Usenet administration on the Internet and consists of five steps: 1) informal discussion, 2) request for formal discussion, 3) formal discussion, 4) voting, and 5) posting of results. The newsgroup can be accessed using any news reader via the Internet.

Dean F. Sittig, PhD; Grand Challenges in Medical Informatics?, Journal of the American Medical Informatics Association, Volume 1, Issue 5, 1 September 1994, Pag

Direct computer-based physician order entry has been the subject of debate for over 20 years. Many sites have implemented systems successfully. Others have failed outright or flirted with disaster, incurring substantial delays, cost overruns, and threatened work actions. The rationale for physician order entry includes process improvement, support of cost-conscious decision making, clinical decision support, and optimization of physicians' time. Barriers to physician order entry result from the changes required in practice patterns, roles within the care team, teaching patterns, and institutional policies. Key ingredients for successful implementation include: the system must be fast and easy to use, the user interface must behave consistently in all situations, the institution must have broad and committed involvement and direction by clinicians prior to implementation, the top leadership of the organization must be committed to the project, and a group of problem solvers and users must meet regularly to work out procedural issues. This article reviews the peer-reviewed scientific literature to present the current state of the art of computer-based physician order entry.

Fuzzy set theory makes it possible to map inexact data, concepts, and events to fuzzy sets via user-defined membership functions. The authors describe a method for (1) robustly estimating the mean and slope of an arbitrary number of data points, (2) developing a set of fuzzy membership functions to classify various properties of heart rate trends, and (3) finding the longest consecutive sequence of heart rate data that fit a particular fuzzy membership function. Preliminary results indicate that fuzzy set theory has significant potential in the development of a clinically robust method for classifying heart rate data, trends, and artifacts.>