OBJECTIVES: We report on findings from a meeting convened by the American College of Medical Informatics (ACMI) to characterize aspects of the patient experience that could be improved using informatics. MATERIALS AND METHODS: The American College of Medical Informatics fellows were invited to share their experiences as patients and suggest informatics approaches that may improve the patient experience. RESULTS: We identified 4 themes: (1) getting the right care, (2) data sharing and data interoperability, (3) guiding low-cost evaluations, and (4) predictive analytics. DISCUSSION: Despite widespread adoption of health IT, patient experiences remain far from optimal. CONCLUSION: The American College of Medical Informatics fellows identified informatics approaches, applications, and research areas that have the potential to improve patient experiences with health care systems.

A panel sponsored by the American College of Medical Informatics (ACMI) at the 2021 AMIA Symposium addressed the provocative question: "Are Electronic Health Records dumbing down clinicians?" After reviewing electronic health record (EHR) development and evolution, the panel discussed how EHR use can impair care delivery. Both suboptimal functionality during EHR use and longer-term effects outside of EHR use can reduce clinicians' efficiencies, reasoning abilities, and knowledge. Panel members explored potential solutions to problems discussed. Progress will require significant engagement from clinician-users, educators, health systems, commercial vendors, regulators, and policy makers. Future EHR systems must become more user-focused and scalable and enable providers to work smarter to deliver improved care.

Dear JAMIA Editors, In their recent article, “Risk prediction of delirium in hospitalized patients using machine learning: an implementation and prospective evaluation study,” Jauk et al implemented and prospectively evaluated the performance of a machine learning algorithm to predict delirium in hospitalized patients using electronic health record (EHR) data available at admission and on the first evening after admission.1 This is an important problem in hospital medicine and neurology where delirium, a preventable condition, is under-recognized and under-treated and often leads to extended lengths of stay, increased health care costs, and acceleration in existing cognitive decline. Jauk et al demonstrated noteworthy accomplishments that will lead to exciting future investigations: (1) integrating the delirium predictive model into a clinical workflow within the EHR, (2) unobtrusively using data captured and documented in the normal care of patients, and (3) evaluating the performance of their model prospectively in the clinical setting. However, this study also illustrates a critical flaw in our approach to applying artificial intelligence and machine learning that prompts the question: what is the “ground truth” on which we are training our models? We should take pause before implementing machine learning algorithms in clinical contexts and assess the underlying classification task of the algorithms. The authors acknowledged a limitation of their study: basing the occurrence of delirium on the presence of International Classification of Disease-Tenth Revision (ICD-10 codes, terms and text © World Health Organization, Third Edition. 2007) codes F05 (“delirium due to known physiological condition” including all subcategories) and F10.4 (“alcohol withdrawal state with delirium”) assigned as diagnoses for the encounter. They recognize that a “lack of clear diagnostic criteria” for delirium “might be one reason why the incidence of delirium according to ICD codes in an administrative database (1.5% in this study) is lower than the one reported in prospective studies (ranging from 10%–40%).” Indeed, in the roadmap to advance delirium research from the Network for Investigation of Delirium: Unifying Scientists (NIDUS), Oh et al describe the need for a refined definition of and a reference standard for diagnosis of delirium.2 However, lack of a clear reference standard for delirium is not enough to explain such a deviation from prior measured incidence rates. In this case, it is apparent that the ground truth missed cases of delirium when it was present. Thus, efforts should have been made to evaluate and improve the ground truth prior to using it to train the predictive models because what algorithms are predicting might be the bias of determining the diagnosis, not the condition itself. Much like how the lack of a gold standard for diagnosis of cancer limits the utility of machine learning algorithms for diagnosing early stage cancer,3 the lack of clear diagnostic criteria to define delirium along with the dependence on the presence or absence of diagnosis codes limit the utility of the machine learning algorithm. If the algorithm performs prospectively as well as it does on the training set, it would only successfully identify cases that would have been coded with a diagnosis of delirium. Defining clinical conditions using available data, or defining “digital phenotypes,” is an art and a science in biomedical informatics. Definitions of clinical conditions have wide variability based on the data used, such as with congestive heart failure. Data beyond ICD codes are needed to improve the positive predictive value for conditions.4 Research support informatics teams have been developed at academic health centers to aid researchers in defining patient cohorts with various clinical conditions based on the best data available. Including different modalities (diagnosis codes, lab values, vital signs, reference to specific symptoms in the notes) of data in the digital phenotype definition process refines the accuracy of the cohort. We see evidence in this study of erroneously depending on diagnosis codes alone to define the digital phenotype of delirium in the comparison of expert nurses’ risk ratings for delirium compared to the calculated risk of the algorithm. There was a wide range of both predicted risk and nursing risk assessments of delirium and there was correlation between the predictive model and expert nursing assessments. However, in this study, 0/33 from the initial nursing assessment evaluation and 2/86 from the second nursing assessment had diagnoses of delirium (1 was correctly identified by the algorithm alone, and 1 was correctly identified by expert nursing alone). The authors expanded the cohort definition of delirium by searching free-text patient summaries for words related to delirium and, if positive, manually checking the cases for evidence of delirium. However, there was still a substantial difference in the incidence rate from this study and benchmark incidence rates. Additionally, the authors recognized that delirium is “not always coded in the participating hospital, and sometimes it is not even mentioned in the discharge summary.” The authors cite a lack of available data in the EHR limiting both the diagnosis of delirium as well as the performance of the prediction models. In particular, if a patient is new to the system, there is a lack of prior data. Even by including data collected during the first day, the lack of prior data posed challenges to the prediction model. This can be attributed to a dependence on structured data (demographic data, diagnosis data, laboratory data, nursing assessments, and procedures). Clinical notes, even in the emergency department setting, are a valuable source of clinically relevant data.5 Natural language processing technologies available today, and constantly improving, can extract phenotypic data from unstructured free-text notes. Such methods could make sufficient data accessible both to improve the accuracy of the digital phenotype of delirium as well as to improve the prediction model, even in those with no prior encounters. This work demonstrated that there are great opportunities to improve the defined digital phenotypes for delirium as well as other conditions to use as a more accurate ground truth in developing prediction algorithms. Natural language processing technologies can extend the search for useful data beyond those coded in the EHR to free-text reports and notes to improve both definition and prediction, but effort is needed to curate and optimize the ground truth we use to train future predictive models. This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. Both authors conceived the correspondence. JR drafted the correspondence and WT reviewed and edited the correspondence. Both authors had final approval of the correspondence and are accountable for all aspects of the work. None declared.

Clinicians often attribute much of their burnout experience to use of the electronic health record, the adoption of which was greatly accelerated by the Health Information Technology for Economic and Clinical Health Act of 2009. That same year, AMIA's Policy Meeting focused on possible unintended consequences associated with rapid implementation of electronic health records, generating 17 potential consequences and 15 recommendations to address them. At the 2020 annual meeting of the American College of Medical Informatics (ACMI), ACMI fellows participated in a modified Delphi process to assess the accuracy of the 2009 predictions and the response to the recommendations. Among the findings, the fellows concluded that the degree of clinician burnout and its contributing factors, such as increased documentation requirements, were significantly underestimated. Conversely, problems related to identify theft and fraud were overestimated. Only 3 of the 15 recommendations were adjudged more than half-addressed.

Founded in 1984, the American College of Medical Informatics (ACMI) is a college of elected Fellows from the United States and abroad who have made significant and sustained contributions to the field of biomedical informatics. On November 4, 2018, the 2018 Cohort of Fellows was introduced to the College and attendees at the American Medical Informatics Association (AMIA) Annual Symposium as well as to the public through Tweets. This article includes the introduction for each Fellow in the 2018 Cohort, which was read by Christopher G. Chute (ACMI President), Suzanne Bakken (ACMI Past President), or William M. Tierney (ACMI President-Elect); a somewhat tongue-in-cheek Tweet created by Gretchen Purcell Jackson or James J. Cimino; and a link to their Journal of theAmerican Medical Informatics Association (JAMIA) and JAMIA Open publications. This is followed by the traditional closing remarks of welcome into ACMI. Gregory L. Alexander,PhD, RN, FAAN, FACMI Potter-Brinton Endowed Professor, Sinclair School of Nursing and Department of Health Management and Informatics, University of Missouri

OBJECTIVE: Efficient, effective health care requires rapid availability of patient information. We designed, implemented, and assessed the impact of a primary care electronic medical record (EMR) in three rural Kenyan health centers. METHOD: Local clinicians identified data required for primary care and public health reporting. We designed paper encounter forms to capture these data in adult medicine, pediatric, and antenatal clinics. Encounter form data were hand-entered into a new primary care module in an existing EMR serving onsite clinics serving patients infected with the human immunodeficiency virus (HIV). Before subsequent visits, Summary Reports were printed containing selected patient data with reminders for needed HIV care. We assessed effects on patient flow and provider work with time-motion studies before implementation and two years later, and we surveyed providers' satisfaction with the EMR. RESULTS: Between September 2008 and December 2011, 72 635 primary care patients were registered and 114 480 encounter forms were completed. During 2011, 32 193 unique patients visited primary care clinics, and encounter forms were completed for all visits. Of 1031 (3.2%) who were HIV-infected, 85% received HIV care. Patient clinic time increased from 37 to 81 min/visit after EMR implementation in one health center and 56 to 106 min/visit in the other. However, outpatient visits to both health centers increased by 85%. Three-quarters of increased time was spent waiting. Despite nearly doubling visits, there was no change in clinical officers' work patterns, but the nurses' and the clerks' patient care time decreased after EMR implementation. Providers were generally satisfied with the EMR but desired additional training. CONCLUSIONS: We successfully implemented a primary care EMR in three rural Kenyan health centers. Patient waiting time was dramatically lengthened while the nurses' and the clerks' patient care time decreased. Long-term use of EMRs in such settings will require changes in culture and workflow.

Over the last 5 years, stimulated by the changing healthcare environment and the Health Information Technology for Economic and Clinical Health (HITECH) Meaningful Use (MU) Electronic Health Record (EHR) Incentive program, EHR adoption has increased remarkably, and there is early evidence that such adoption has resulted in healthcare safety and quality benefits.1,2 However, with this broad adoption, many clinicians are voicing concerns that EHR use has had unintended clinical consequences, including reduced time for patient-clinician interaction,3 new and burdensome data entry tasks being transferred to front-line clinicians,4,5 and lengthened clinician workdays.6–8 Additionally, interoperability between different EHR systems has languished despite large efforts towards that goal.9,10 These challenges are contributing to physicians’ decreased satisfaction with their work lives.11–13 In professional journals,14 press reports,15–17 on wards, and in clinics, we have heard of the difficulties that the transition from paper records to EHRs has created.18 As a result, clinicians are seeking help to get through their work days, which often extend into evenings devoted to writing notes. Examples of comments we have received from clinicians and patients include: “Computers always make things faster and cheaper. Not this time,” and “My doctor pays more attention to the computer than to me.”

OBJECTIVE: Little evidence exists on effective interventions to integrate HIV-care guidelines into practices within developing countries. This study tested the hypothesis that clinical summaries with computer-generated reminders could improve clinicians' compliance with CD4 testing guidelines in the resource-limited setting of sub-Saharan Africa. DESIGN: A prospective comparative study of two randomly selected outpatient adult HIV clinics in western Kenya. Printed summaries with reminders for overdue CD4 tests were made available to clinicians in the intervention clinic but not in the control clinic. MEASUREMENTS: Changes in order rates for overdue CD4 tests were compared between and within the two clinics. RESULTS: The computerized reminder system identified 717 encounters (21%) with overdue CD4 tests. Analysis by study assignment (regardless of summaries being printed or not) revealed that with computer-generated reminders, CD4 order rates were significantly higher in the intervention clinic compared to the control clinic (53% vs 38%, OR = 1.80, CI 1.34 to 2.42, p < 0.0001). When comparison was restricted to encounters where summaries with reminders were printed, order rates in intervention clinic were even higher (63%). The intervention clinic increased CD4 ordering from 42% before reminders to 63% with reminders (50% increase, OR = 2.32, CI 1.67 to 3.22, p < 0.0001), compared to control clinic with only 8% increase from prestudy baseline (CI 0.83 to 1.46, p = 0.51). Limitations Evaluation was conducted at two clinics in a single institution. CONCLUSIONS: Clinical summaries with computer-generated reminders significantly improved clinician compliance with CD4 testing guidelines in the resource-limited setting of sub-Saharan Africa. This technology can have broad applicability to improve quality of HIV care in these settings.

Current models for implementing electronic health records (EHRs) in resource-limited settings may not be scalable because they fail to address human-resource and cost constraints. This paper describes an implementation model which relies on shared responsibility between local sites and an external three-pronged support infrastructure consisting of: (1) a national technical expertise center, (2) an implementer's community, and (3) a developer's community. This model was used to implement an open-source EHR in three Ugandan HIV-clinics. Pre-post time-motion study at one site revealed that Primary Care Providers spent a third less time in direct and indirect care of patients (p<0.001) and 40% more time on personal activities (p=0.09) after EHRs implementation. Time spent by previously enrolled patients with non-clinician staff fell by half (p=0.004) and with pharmacy by 63% (p<0.001). Surveyed providers were highly satisfied with the EHRs and its support infrastructure. This model offers a viable approach for broadly implementing EHRs in resource-limited settings.

OBJECTIVE: The AMPATH program is a leading initiative in rural Kenya providing healthcare services to combat HIV. Malnutrition and food insecurity are common among AMPATH patients and the Nutritional Information System (NIS) was designed, with cross-functional collaboration between engineering and medical communities, as a comprehensive electronic system to record and assist in effective food distribution in a region with poor infrastructure. DESIGN: The NIS was designed modularly to support the urgent need of a system for the growing food distribution program. The system manages the ordering, storage, packing, shipping, and distribution of fresh produce from AMPATH farms and dry food supplements from the World Food Programme (WFP) and U.S. Agency for International Development (USAID) based on nutritionists' prescriptions for food supplements. Additionally, the system also records details of food distributed to support future studies. MEASUREMENTS: Patients fed weekly, patient visits per month. RESULTS: With inception of the NIS, the AMPATH food distribution program was able to support 30,000 persons fed weekly, up from 2,000 persons. Patient visits per month also saw a marked increase. CONCLUSION: The NIS' modular design and frequent, effective interactions between developers and users has positively affected the design, implementation, support, and modifications of the NIS. It demonstrates the success of collaboration between engineering and medical communities, and more importantly the feasibility for technology readily available in a modern country to contribute to healthcare delivery in developing countries like Kenya and other parts of sub-Saharan Africa.

Providing quality health care requires access to continuous patient data that developing countries often lack. A panel of medical informatics specialists, clinical human immunodeficiency virus (HIV) specialists, and program managers suggests a minimum data set for supporting the management and monitoring of patients with HIV and their care programs in developing countries. The proposed minimum data set consists of data for registration and scheduling, monitoring and improving practice management, and describing clinical encounters and clinical care. Data should be numeric or coded using standard definitions and minimal free text. To enhance accuracy, efficiency, and availability, data should be recorded electronically by those generating them. Data elements must be sufficiently detailed to support clinical algorithms/guidelines and aggregation into broader categories for consumption by higher level users (e.g., national and international health care agencies). The proposed minimum data set will evolve over time as funding increases, care protocols change, and additional tests and treatments become available for HIV-infected patients in developing countries.

OBJECTIVE: Authors evaluated whether displaying context sensitive links to infrequently accessed educational materials and patient information via the user interface of an inpatient computerized care provider order entry (CPOE) system would affect access rates to the materials. DESIGN: The CPOE of Vanderbilt University Hospital (VUH) included "baseline" clinical decision support advice for safety and quality. Authors augmented this with seven new primarily educational decision support features. A prospective, randomized, controlled trial compared clinicians' utilization rates for the new materials via two interfaces. Control subjects could access study-related decision support from a menu in the standard CPOE interface. Intervention subjects received active notification when study-related decision support was available through context sensitive, visibly highlighted, selectable hyperlinks. MEASUREMENTS: Rates of opportunities to access and utilization of study-related decision support materials from April 1999 through March 2000 on seven VUH Internal Medicine wards. RESULTS: During 4,466 intervention subject-days, there were 240,504 (53.9/subject-day) opportunities for study-related decision support, while during 3,397 control subject-days, there were 178,235 (52.5/subject-day) opportunities for such decision support, respectively (p = 0.11). Individual intervention subjects accessed the decision support features at least once on 3.8% of subject-days logged on (278 responses); controls accessed it at least once on 0.6% of subject-days (18 responses), with a response rate ratio adjusted for decision support frequency of 9.17 (95% confidence interval 4.6-18, p < 0.0005). On average, intervention subjects accessed study-related decision support materials once every 16 days individually and once every 1.26 days in aggregate. CONCLUSION: Highlighting availability of context-sensitive educational materials and patient information through visible hyperlinks significantly increased utilization rates for study-related decision support when compared to "standard" VUH CPOE methods, although absolute response rates were low.

Two papers in this issue of JAMIA discuss computerized physician order entry (CPOE) and a third one discusses patient clinical information systems (PCISs), which often include CPOE. The first paper, by Ash and colleagues, simply reports the rate at which U.S. hospitals and their care providers are adopting physician order entry systems.1 The other two challenge the current push toward rapid adoption of CPOE and PCIS in the health care industry—Berger and Kichak,2 by challenging the evidence base for the push, and Ash et al.,3 by calling attention to many failure points that occur when rigid computer system designs meet the reality of really complex clinical systems. Before commenting on these papers, we should confess our long-term infatuation with computers and a 30-year conviction that computers could be the “chicken soup” for many illnesses of the health care system. We proved that computer reminders systems are chicken soup for preventive care in a series of studies starting in 1976.4–7 Then, during the early 1980s, we spent our nights and weekends and as much time as we could scrape from our workday writing, testing, implementing, and studying software for what would become the Medical Gopher,8 the first PC-based order entry system used, and studied, in outpatient care.9–11 More years were required to tune and adapt this system, born in an outpatient setting, for an inpatient service. Then we performed the first, and what may be the only, randomized trial of CPOE in the hospital and proved that our Medical Gopher order entry system is chicken soup to hospital inefficiency. It reduced the cost of care and improved the workflow among CPOE users by 13% compared with the control group who used the traditional paper orders. An example of its benefit to work flow improvement is the 12-fold reduction in the delay from writing admission orders to the execution of those orders: from an average of six hours to 30 minutes. And physicians liked it. Because the hospital liked it too, we then extended it to include all hospital services. This hospital (perhaps coincidentally) now has the lowest mortality rate and the third lowest costs of all hospitals in the University Hospital Consortium (UHC).11 While on the subject of this controlled trial of the Medical Gopher,12 we have to quibble with Berger and Kichak's statement that it included only one internal medicine service. In fact, it included all six internal medicine services at the hospital, half of which used the computer order entry system and half of which did not. As we published papers on this system in the late 1980s and 1990s, some reviewers criticized the work as being irrelevant because no one else used or ever would use CPOE. So it tickles us to read that as many as 13% of U.S. hospitals now have CPOE.3 The principal argument for the push to adopt CPOE has been the promise of lifesaving benefits. While we believe CPOE has great potential to improve the health care process, we have to agree with Berger and Kichak's position that a convincing case for lifesaving benefits has not been made. As Berger and Kichak point out, the keystone for these arguments—namely, the Institute of Medicine's (IOM's) claim of 44,000 to 98,000 deaths due to medical errors—crumbles on close inspection. The 98,000 figure was extrapolated from the 173 deaths (13.6%) among selected New York patients who were hospitalized in 1984 and had an adverse event. The extrapolation assumes that none of these fairly sick hospitalized patients would have died in the absence of the adverse event, and ignores the fact that the patients with adverse events had a death rate no different than the death rate (13.8%) of the target population from which they were drawn.13 Any arbitrary criteria—such as being assigned a hospital number ending in the digit 3—used to select patients from the same target population would have led to the same high death rate. Leape has suggested that one reason why airplanes are safer than hospitals is that the pilot goes down with the plane.14 What he apparently fails to realize is that (1) hospitals are substantially more complex than airplanes (with more people and moving parts and systems that need to integrate), and (2), for a large number of disastrously ill patients, the plane is already on its way down when the physician-pilot climbs on board. Errors happen more often in the most complex situations, where mortality risk is highest to begin with. Berger and Kichak also remind us that new technology is never completely virtuous. CPOE eliminates illegible orders and provides opportunities for better ordering, but computer systems also introduce errors of their own. As we were writing this editorial, the United States Pharmacopeia (USP) published a report to reinforce this point. Based on one year's worth of reports from 480 U.S. hospitals, the USP found that 8.2% of potentially harmful medication ordering errors arise from computer order entry errors.15 (The majority of these errors were likely due to medication orders entered by a pharmacist or nurse.) A slip of the mouse on a computer menu can lead to an order for the right medication for the wrong patient. Ash et al. also testify to the existence of this problem in their paper.3 This risk is unlikely to outweigh the other benefits of CPOE but should encourage developers to build traps for catching and preventing such errors. Finally, there is the question of what added safety benefits arise from having one health care professional (e.g., the physician) instead of another (e.g., the pharmacist) enter an order when the computer system can apply exactly the same safety checks regardless of who enters the order. One might say physician order entry is needed to solve the prescription legibility problem, but when examined closely, this problem does not loom as large as assumed. Illegibility is not even listed in the top causes of medication errors in any of the large studies—probably because pharmacists call physicians to clarify illegible prescriptions.16,17 Further, the legibility of physicians' writing is better than its reputation.18–19 Finally, the problem of ambiguous medication names and doses can be ameliorated by Joint Commission on Accreditation of Healthcare Organization (JCAHO) regulations that will banish certain abbreviations and prescription writing conventions.20 Ash and colleagues raise a host of similar issues related to the ability of rigid computer systems to tame and serve the enormously complex and time-critical processes that churn within health care institutions.3 Their report should be required reading by every health care system institution's CEO and CIO. Ash et al. remind us that highly structured clinical data are usually more difficult to enter, and almost always more difficult to read and to digest, than human-crafted text. Confirming their position, we noticed that our physicians choose the last hospital discharge summary, not the data flow sheet, as the first thing to read when reviewing a patient new to them in our computer system. The use of a structured entry form does more than convert what the provider would have said as narrative into computer-understandable content. Depending on its design, a structured questionnaire may also inhibit the recording of subtleties and details that would have flowed naturally as narrative. Structured entry forms may also demand more information than would have been recorded as free text. This can be bad or good, depending on the relevance of the extra questions to the patient at hand. Our profession lacks evidence about the value and predictive content of most of the discrete history and physical elements it collects. The Ottawa ankle rule,21 whose authors analyzed more than 50 candidate variables to find the eight that predict ankle fractures, illustrates the advantage of careful study of the value of clinical data collection. But since only a tiny fraction of history and clinical findings has had such careful study, computer systems that gather history and physical information through structured forms are prone to asking too many questions of unknown informational value. In addition, Ash and colleagues teach us about the interruptive nature of the health care process.3 Health care workers use computers in short bursts and flit among computers like honeybees among flowers. This usage model is quite different from the dominant model in the business world, where users log onto their computers once at the beginning of the day and stay with the same machine all day, making a 1-minute log-in tolerable. Health care workers may log into a different computer 10 times an hour and cannot afford a 1-minute or even a 20-second delay every time they log in. But CIOs and security officers are not always sensitive to this difference, and thus may employ the same slow operating system log-ons used in business settings, with disastrous effects on user efficiency. Finally, Ash and colleagues call attention to decision support overload.3 Too many nonspecific and repetitive reminders are the moral equivalent of e-mail “spam” and cause the same justified annoyance to the recipient. Further, such overload will dull the physician's attention to the less common reminders that really matter. The antidotes to excessive and inappropriate reminders are strict constraints on what reminder rules are adopted, i.e., rules that have a strong evidence base, that can be decided based on the kinds of information that the computer carries, and that are vetted by a balanced committee of the providers who get the reminders. Alternatively, physicians could decide what things the computer should remind them to do, turning “computer” reminders into “self” reminders. No study has shown any direct health outcome benefit from CPOE, and we doubt that CPOE (order entry by the physician per se) systems will produce lifesaving benefits that cannot be delivered by other computer processes (e.g., checking on drug dosages when pharmacists enter the orders or reminders delivered to physicians through other mechanisms).22 On the other hand, CPOE systems definitely can have large and important benefits on institutional efficiency and costs. Our study of CPOE showed a 13% improvement in care efficiency, but a zero difference in measures of patient outcomes either during or after the hospital stay. Others have shown similar results. Eighty percent of all care costs are initiated by a physician order23 and CPOE systems can induce more cost-effective choices among their physician users. CPOE order menus can guide providers to the more cost-efficient test and treatment options by making it easier to choose them. CPOEs can pop up counter-detailing information about costs and better alternatives as the provider makes his or her choices.10,24 Through rules and templates, CPOE systems can focus providers on the least expensive choice of medication within a class or the one for which the patient will have to make the least copay.25 The CPOE system can even pick the most appropriate cardiac stress test for a given patient. CPOE systems can also improve and simplify compliance with regulatory requirements, such as the management of short-stay patients, the implementation of medical necessity rules, and more. We have more rules in our system for controlling costs and facilitating regulatory compliance than for improving quality. Cost issues are crucial to our inner-city county hospital, which incurred a $40 million deficit this year, even with all of our efforts. These dollars are ever more important for safety net hospitals such as Wishard in a world of reduced reimbursement. CPOE systems can reduce unnecessary repeat testing26 and the delays between writing and completing orders. They can also reduce labor costs directly by reducing the time spent by nursing, pharmacy, and other ancillary services on callbacks to clarify orders and by eliminating the personnel time of transcribing orders. So, health care institutions have much to gain in efficiency and cost savings from CPOE systems. Sadly, this is a win–lose game. CPOE systems generally cause physicians to lose efficiency. It takes physicians longer to enter orders into computers than to write them on paper, and this can mean 30 or more extra minutes out of the physician's day11,27—time that cannot be used caring for patients. So the physician is being asked to pay the price for the efficiency and/or cost reductions that are enjoyed by the institution or payer. But because the imbalance is an economic one and the cost to the physicians is much less than the potential gain to the institutions and/or payers, many good options exist to make this a win–win game. First, institutions should not try to extract all possible communication efficiencies by forcing extreme or unfamiliar coding on the physicians, for the reasons articulated by Ash and colleagues.3 The menus from which physicians choose their orders should correspond to what physicians now name as concepts (e.g., oxygen orders or ampicillin caps), not items defined by the inventory system (e.g., ampicillin 500-mg caps in 250-mg capsule bottles from Wyeth) or the billing system (e.g., oxygen 30% ventilator mask). Physicians should be allowed to select an order called “miscellaneous medication” and type in the name of the medication they want, when they cannot easily find the order in the menu displayed. And physicians should be allowed to write their instructions (Sig) for an order as narrative (e.g., 500 mg 4 × per day × 10 days for earache) as they do now and/or edit the default instructions provided by the CPOE system. Institutions should not push for the capture of data that are not a part of conventional order-writing just to make life easier for their ancillary services, because this adds to the physician's time burden. We have seen CPOE systems that demand the provider enter the start and end time as a specific time and date, and others that require the writing of sliding scale insulin as four or five separate orders instead of a single insulin order with the sliding scale given in the Sig. Yes, making such compromises will require that the institution invest some personnel time coding some physician-entered orders, but this is a fair quid pro quo for a physician commitment to CPOE. Furthermore, human reviews of some kinds of orders, for example, medication orders, will continue to be needed for safety reasons. Second, the institutions and/or payers who gain the economic advantage of CPOE systems should recognize the time cost to physicians and provide economic adjustments. One care system has provided an incentive for its employed physicians by adjusting downward the number of patients the system expects physicians to see each day. We can imagine other approaches, such as higher reimbursement rates to physicians who agree to write all of their inpatient orders through an order entry system. Third, vendors and institutions have to be more focused and more inventive in simplifying physician order entry to eliminate (or at least minimize) the time disadvantage. The answer may lie with “bigger,” more protocolized orders—e.g., a single order that could/would request the institution's preferred statin drug and also order appropriate testing for that drug.28 Or perhaps for the occasional physician user, an option could exist for real-time dictation of orders to an operator who enters the orders into a remote computer, which echoes these entered orders back to the physician's computer where he or she can react to reminders and alerts and confirm or adjust the orders. Such a system has been implemented.29 We were glad to see the papers by Berger and Kichak2 and Ash et al.3 because they bring some needed balance to the current excessive expectations about CPOE and computer decision support and will lead to more successful and sustainable PCIS and health care outcomes in the long run. The political energy that is now being directed toward CPOE should instead be focused on encouraging faster improvements in the National Health Information Infrastructure (NHII)30 and deployment of useful clinical data repositories. The NHII is a prerequisite to affordable health care information systems and such repositories are a godsend to physicians and necessary for useful decision support. These developments are also critical if CPOE and PCIS are to achieve their promise of enhancing the quality of health care while controlling its costs.

The authors describe a research group that supports the needs of investigators seeking data from an electronic medical record system. Since its creation in 1972, the Regenstrief Medical Records System has captured and stored more than 350 million discrete coded observations on two million patients. This repository has become a central data source for prospective and retrospective research. It is accessed by six data analysts--working closely with the institutional review board--who provide investigators with timely and accurate data while protecting patient and provider privacy and confidentiality. From January 1, 1999, to July 31, 2002, data analysts tracked their activities involving 47,559 hours of work predominantly for physicians (54%). While data retrieval (36%) and analysis (25%) were primary activities, data analysts also actively collaborated with researchers. Primary objectives of data provided to investigators were to address disease-specific (35.4%) and drug-related (12.2%) questions, support guideline implementation (13.1%), and probe various aspects of clinical epidemiology (5.7%). Outcomes of these endeavors included 117 grants (including 300,000 US dollars per year salary support for data analysts) and 139 papers in peer-reviewed journals by investigators who rated the support provided by data analysts as extremely valuable.

The Institute of Medicine has declared electronic medical records to be an essential technology for health care1 and a necessary tool for improving patient safety2 and the quality of care.3 To date, comprehensive computer-based patient records that serve these functions are uncommonly used in developed countries,4 and are rare to nonexistent in the developing world. This gulf has been termed the digital divide5 and even technological apartheid6 where the simplest technology is not available to promote health care delivery, patient outcomes, and public health. We have reported previously the conceptualization and initial development of the Mosoriot Medical Record System (MMRS),7 an electronic medical record system supporting a primary care health center in rural Kenya. In this article, we report the implementation of the MMRS within the Mosoriot Rural Health Centre (MRHC) as the sole means for recording clinical data. We emphasize the technical aspects of data capture and storage, describe data from the first 10,000 visit records, and report the results of a formal evaluation of the impact of MMRS on patterns of health care delivery within the MRHC. This study was approved by Indiana University's Institutional Review Board and the Ethics Committee of the Moi University College of Health Sciences (MUCHS). There is a long-term collaboration between Indiana University and MUCHS.8,9 MUCHS uses a number of rural health centers in western Kenya as part of its medical education and public health research programs. The MRHC is one of these centers, located approximately 25 km (15 miles) southwest of Eldoret, Kenya's fifth largest city. This region is a highland plateau characterized by limited availability of essential resources such as potable water, sewerage, and paper. Electrical power and land-based telephone lines, when available at all, are unreliable. Cellular telephones are widely available and reliable, but they are relatively expensive and, thus, beyond the reach of most Kenyans. The economic infrastructure is mainly subsistence farming with significant poverty and unemployment. The MRHC is maintained by the Kenyan Ministry of Health to provide primary and emergency care to a surrounding agrarian population of approximately 40,000 persons who mainly live in small villages. Although these villages have traditional (i.e., non-Western) healers and midwives, all state-sponsored health care is delivered in the MRHC. Except for severe emergencies such as major trauma, persons without financial resources who live in the MRHC's catchment area must be referred by the MRHC to receive care elsewhere in the Kenyan health care system. The MRHC contains a number of primary and urgent care clinics: antenatal, child welfare (under 5 years old), pediatric, adult medicine, family planning, and sexually transmitted infections. There also is a small inpatient unit (under 20 beds) in which patients requiring monitoring and more extended care can be hospitalized for brief intervals. Patients needing longer inpatient stays or more sophisticated care are referred to the Moi Teaching and Referral Hospital in Eldoret that is staffed by faculty from MUCHS. Before the implementation of the MMRS, patients visiting the MRHC stopped at the check-in window where the visit was recorded, and the patient was given a sequential number for that visit that year. There was no permanent patient registry or unique identifier. Then, depending on each patient's age and clinical problem (or prior appointment), the patient was directed to the appropriate clinic. Care at the MRHC is provided by nurses and their assistants with oversight by a single, nonphysician clinical officer. Because the MRHC is a site for education of medical, nursing, and public health students from multiple Kenyan institutions, students often participate in providing care. When diagnostic tests are needed, the patient is directed to a small laboratory where blood smears, urinalyses, and a small number of serologic tests are performed. A basic x-ray unit can take simple chest and bone radiographs. There is a pharmacy that contains bulk quantities of a very small number of drugs, mostly antibiotics and analgesics. Before installation of the MMRS, records for MRHC visits were kept in logbooks maintained in both the registration office and each clinic. Identical (and duplicative) information for each visit was entered by hand into the logbook of each clinic the patient visited and consisted of the visit number (numbers are recycled at the beginning of each year), the patient's name, the chief complaint, a final diagnosis, and treatment given or prescribed. A single file cabinet holds more than 30 years of medical records for the entire facility (Figure 1, located in an online data supplement at ). The nurses and clinical officers record additional clinical notes (diagnoses, test results, etc.) in small booklets that each individual patient is required to purchase each year for $0.25 (US). The patients take these booklets home along with any radiographs that are taken. The patients are expected to bring these booklets to each visit, the value of which is limited by their cost (which can be expensive to the typically large Kenyan families with little or no income) and patients' sometimes forgetting them. The initial conceptualization and development of the MMRS have been described in detail elsewhere.7 Briefly, it is a modular system comprised of a Registration Module, a paper encounter form, a Data Entry module, a Reporting Module, and a Data Dictionary. When a new patient presents to the check-in window, he or she is registered into the MMRS and given a plastic card on which his or her name and MMRS number are recorded (Figure 2, located in an online data supplement at ). Patients proudly carry their cards identifying them as “members” of the MRHC. At all subsequent visits, the patient is expected to present his or her identification (ID) card, and the MMRS number is handwritten on a blank encounter form (Figure 3). If a previously registered patient does not have his or her card, the MMRS registration system provides a name lookup option. Paper encounter form on which health care providers enter patient data. Each patient then is given an encounter form and directed to the appropriate clinic. Nurses in each clinic record clinical information on the encounter form rather than writing it in the clinic or laboratory logbooks or the patients' booklets. Laboratory technicians do the same for patients undergoing diagnostic tests. The encounter form is designed to require minimal writing using check boxes whenever possible. After visits to the clinic(s), laboratory, pharmacy, and financial office, each patient is supposed to return to the check-out window and present the encounter form to the check-out clerk. The clerk then enters encounter form data into the MMRS and returns the encounter form to the patient to take home in lieu of providers' writing in the patient's booklet. Figure 4 shows the latest version of the MUCHs' main menu screen, and Figure 5 (located in an online data supplement at ) shows the registration screen. Figure 6 shows the encounter form data entry form, with the antenatal clinic screen displayed. The data model uses the patient and date as the unique identifiers of the visit, the basic unit of observation. All clinic, test, pharmacy, and charge data are linked to the visit. Main menu screen for the Mosoriot Medical Record System. Encounter form data entry screen, with the antenatal clinic tab selected. The MRHC gets power from the local electrical grid, which obtains 90% of its power from hydroelectric dams. During dry spells, the water level behind these dams drops drastically, which results in power rationing: certain regions of the country and services are deprived of electric power on a rotating basis so that the reserve can be used for important services such as hospitals and security facilities. The MRHC is in a rural area considered not large enough to warrant receiving continuous power supply. Therefore, to ensure sustainability and continued operation of the MMRS, we provided multiple backup systems: An uninterruptible power source (UPS) battery A solar-powered system A gasoline-powered generator A paper backup system (paper registration and encounter forms that can be back-entered into the system when power is restored) With the use of a power inverter (converts DC to AC), the solar system is connected to the main power source via two large batteries so that the system can use either the electric grid or the solar system. The computer system including the identification label printer and the report printer use the above cascade of power sources. Whenever there is main grid failure, the UPS automatically takes over. If the local grid is not restored before the UPS fails (approximately 30 minutes), an alarm sounds, after which a clerk flips a switch, and the system begins drawing power from the solar batteries that last approximately four hours. The choice of the backup solar power system was made because Kenya has sunlight for more than 90% of the day, which, at Mosoriot (located near the equator), is 12 hours long every day of the year. An exception is during the wet season, which is highly variable, when the amount of sunlight is significantly decreased. Fortunately, these are the seasons when the power rationing is usually not in effect because the water level has increased. The solar system is reasonably inexpensive (approximately $1,000 US). Because the MMRS was turned on in February of 2001, there has been no loss of data because of a lack of electricity. The gasoline-powered generator was never used, so it was removed. There is no Kenyan unique identifier equivalent to the American social security number. Therefore, we had to create an MMRS number for each patient using the smallest number of data fields that uniquely identify Kenyans. After consultation with MUCHS faculty and MRHC staff, we decided on the following fields: First name (usually Western) Middle name (usually Kenyan) Last name (often shared by many persons in one's village) Mother's first name Father's first name (not really required technically but required socially to maintain the status of Kenyan men) Village (whose names are not unique) Location (equivalent to a US county) Sublocation (equivalent to a US township) The foundation of the MMRS is the data dictionary. As shown in Figure 7, which is located in an online data supplement at , the dictionary contains a list of terms, reference terms (that allow for multiple synonyms for each term), the term type (diagnosis, test, drug, or treatment), the clinical system to which the term belongs (e.g., organ system, drug class), its ICD-10 code (for diagnoses only), a text description, and a charge for each test and drug. The dictionary has been updated continually as clinicians record new (less common) diagnoses, drugs, and treatments. The data dictionary also has been increased by adding terms the MRHC's managers have requested to generate reports to aid them in their management activities. Before the MMRS could completely replace the MRHC's paper-based record system, the clinicians and staff had to be convinced that the data were secure. such was in where the health care providers and had little or no computer for implementation to and to any that the system the paper system had to be and the MMRS had to the sole source of medical records for MRHC. In to data is but there were no the use of electronic data in a country with no prior with them. We to data security and All to the MMRS are to data for is limited to aspects of the MMRS for which they are a day, the MMRS automatically its entire to a At the of the day when the MMRS computer is the entire is a The backup is home every by the MRHC (i.e., a the system a of the entire on a and it on his computer at MUCHS. Figure 4 shows the reports that are into the of them are required by the Ministry of Health (e.g., of visits by clinic, and an reports can be at the of MRHC's and to aid in their for and MRHC. before the MMRS, the MRHC could not the amount of care required by the Ministry of that is provided to 5 years and with sexually transmitted The MRHC is also using MMRS reports to drug February we a MMRS at MRHC. During the all MRHC clinicians and staff had computer to them with and to their and of the day that the system was turned the local power grid was because a had on the power during the The backup UPS and solar the 20 the MMRS has been for for a of a card, and of the solar data were on because the paper registration and encounter forms were and data were entered and to a medical record system for the developing that is relatively inexpensive and most to be we designed the MMRS to on a single by the same clerk who had been for check-in and check-out the paper-based system. as patients in and were registered into the MMRS, the clerk at the check-in window that she could not and check in patients also data from the encounter forms as patients A single window and clerk could not be used for both check-in and Therefore, because patients in and and them encounter forms were not entered into the system. As this not and the of used encounter forms Therefore, we the system by a computer to the MMRS computer using a between their and a check-out clerk who was for encounter form data. such a clerk approximately US The entire MMRS on the the computer to the on the to which the and are the paper-based record system, the patients after receiving their and visiting the financial office, their patient information booklets with them. During the first of MMRS patients often their encounter forms with their data from entered into the by the and clinic staff the number of encounter forms but the problem additional were to patient the check-out window, including and a that them to the check-out window, adding in both and and the check-in clerk the patient's MRHC card at the of to be by the check-out clerk after the encounter form is mostly lack of computer the data entry had using a to the encounter form data entry screen. the data entry screen was with that the clinic that the patient site within the MRHC such as the laboratory or pharmacy, or a (e.g., Each tab a that contains all of the fields to that can be either by (e.g., for or with a also for of the MMRS to new Figure 5 shows the tab and data entry fields for MRHC's clinic. In we name lookup for data in which the data entered must be a dictionary name lookup and the for the screen has data entry from encounter forms to An electronic medical record system can health care providers which, in can and they the system. We the of the MMRS on at the MRHC by formal before and after was into and final of the MMRS as the medical We during the and final Kenyan research assistants digital assistants in this the health care and medical record the financial The was in When a research first following a he or she a When the an as to an was entered into his or her which it a beginning to the When it to the research the he or she recorded the into one of the of When the the research entered a new into the which an to the and a beginning to the were between the and the research these were with laboratory, pharmacy, and health care the were with with staff, for and the were patient with staff, for writing and The was to a of all nurses and clinical health care providers for one day, and all medical record technicians for multiple Patients were from the they into the MRHC they and the in each of was recorded, along with the in the MRHC. The research the first patient the MRHC that patient he or she the and then the patient the MRHC The health care providers and were usually for the entire they in the MRHC on the day of observation. The of the providers' and between four and hours. were for two each during the and for at a during the these the providers' and were as a of all for each the or between the and were using because of the providers and between the and February and 2, 2001, individual patients had been registered into the MMRS, of visited the MRHC 10,000 As shown in Figure the number of visits with encounter form data from during the first to by the of 2001, the data entry clerk has entered encounter form data for more than of all The in capture of encounter forms when in the MRHC was by a patients to the check-out window where they turned in their encounter rather than them During this the MRHC staff also used the paper-based system, writing patients' diagnoses, and in their logbooks and writing notes in patients' booklets described In 2001, the MRHC and the logbooks from the check-in office and all after which the MMRS the MRHC's sole medical Encounter forms entered into the Mosoriot Medical Record System by for the first the patients the first 10,000 visits, were and were for the was not age was years to The age for patients the adult clinic was shows the with the adult by child welfare (under age and shows the diagnoses and the and shows the tests performed. The were and of during the First 10,000 with Data in the Mosoriot Medical Record System visits are to the antenatal and family where are not recorded because there is technically no of during the First 10,000 with Data in the Mosoriot Medical Record System visits are to the antenatal and family where are not recorded because there is technically no and in the First 10,000 with Data in the Mosoriot Medical Record System of without by a blood and in the First 10,000 with Data in the Mosoriot Medical Record System of without by a blood by the Laboratory and on the First 10,000 with Data in the Mosoriot Medical Record System Laboratory serologic test for test for by the Laboratory and on the First 10,000 with Data in the Mosoriot Medical Record System Laboratory serologic test for test for The first study was in 2001, after of the MMRS but before it was used to The study was in after the initial implementation of the MMRS and after the paper logbooks had been for two During the the research assistants recorded for for health care and for During the final the research assistants recorded for for health care and for As shown in during the using the MMRS, patients with their 12 before the and more of visits from to which was not of of The in was in the of the health care where with patients by from a to a of their also two with staff and their in During the also two with staff and their also one of the writing reports and more for information during the Because of the small of this last was There is additional of the impact of the MMRS on care at the MRHC. The paper logbooks shown in Figure have been because they were never data MRHC patient care are from the In the MRHC's two patterns of care on MMRS there was a of sexually transmitted in one The a of nurses to also a lack of child in nurses to this as where 20 were in the primary and were in the February In the of the MRHC reports that he must reports to the Kenyan Ministry of Health for the number of patients with and the number of Before the MMRS, these reports by hand from the MRHC logbooks a clerk two they take This has the of the MRHC to two medical records to in the MRHC where they are staff in the Ministry of Health the MRHC number one all Kenyan health centers, the first that this has the of Health and the information officers have visited the MRHC to study the MMRS, to it the information system for all its health centers February We have shown that the can be by an inexpensive and electronic medical record system. The to its and continued use by the MRHC's staff, and patients was the to local when the system. This was by the MRHC clinicians and staff in the initial and of the developed a of and of their electronic medical record system, a system that is unique in Kenya and all of to necessary for the initial and of the MMRS, was the that it was developed and maintained mostly by and technicians from MUCHS. implementation of the MMRS was simple MRHC staff had to be convinced that on the additional of a new electronic medical record system and the it required in in the and of care. This was during the initial implementation when both the electronic and paper record were We also had to in the and implementation of the MMRS as not of initial we that a single medical record clerk could both patients and enter data from the encounter This could not be so we had to a computer and a data entry clerk. Although this made the system more expensive and its sustainability in MRHC and implementation at health centers, we had to the MMRS was and developed by an of medical with Kenyan faculty and staff, it was to the system be used or its impact on health care delivery and are to any to the The MMRS the of health care in the MRHC. Patients with health care After the the providers that before the MMRS each clinic had a logbook in which information to that entered into the registration logbook was In each patient had a small in which patient notes were in With the MMRS, the logbooks and the patient booklets were both by the MMRS encounter form which used mostly check boxes to information and which, at the of the visit, was to the patient to take Patients also and their visit to the MRHC was after implementation of the Health care providers and clinical also with patients and staff and had more for activities. for health care the MMRS also a that the managers of the MRHC could for additional (e.g., patient additional patients but writing reports and with the MMRS was for it was in terms of reports for the Kenyan Ministry of We that the MMRS and can be used to health care in developing can and managers for the care they and more for medical and financial identifying patients and care or are health centers in developing can limited resources activities. limited most developing have and public health such as that are mostly at and health medical record such as the MMRS can and these by identify both appropriate for these and providing data. A and electronic medical record system can be and in a developing The to its be its The more such a system is used to care and as a tool for research and development the the that the of implementation and be by both financial and in the health of the

OBJECTIVE: Direct physician order entry (POE) offers many potential benefits, but evidence suggests that POE requires substantially more time than traditional paper-based ordering methods. The Medical Gopher is a well-accepted system for direct POE that has been in use for more than 15 years. The authors hypothesized that physicians using the Gopher would not spend any more time writing orders than physicians using paper-based methods. DESIGN: A randomized controlled trial of POE using the Medical Gopher system in 11 primary care internal medicine practices. MEASUREMENTS: The authors collected detailed time use data using time motion studies of the physicians and surveyed their opinions about the POE system. RESULTS: The authors found that physicians using the Gopher spent 2.2 min more per patient overall, but when duplicative and administrative tasks were taken into account, physicians were found to have spent only 0.43 min more per patient. With experience, the order entry time fell by 3.73 min per patient. The survey revealed that the physicians believed that the system improved their patient care and wanted the Gopher to continue to be available in their practices. CONCLUSIONS: Little extra time, if any, was required for physicians to use the POE system. With experience in its use, physicians may even save time while enjoying the many benefits of POE.

Many adults with cancer are not enrolled in clinical trials because caregivers do not have the time to match the patient's clinical findings with varying eligibility criteria associated with multiple trials for which the patient might be eligible. The authors developed a point-of-use portable decision support tool (DS-TRIEL) to automate this matching process. The support tool consists of a hand-held computer with a programmable relational database. A two-level hierarchic decision framework was used for the identification of eligible subjects for two open breast cancer clinical trials. The hand-held computer also provides protocol consent forms and schemas to further help the busy oncologist. This decision support tool and the decision framework on which it is based could be used for multiple trials and different cancer sites.

OBJECTIVE: To measure the effect of computer-based outpatient prescription writing by internal medicine physicians on pharmacist work patterns. DESIGN: Work sampling at a hospital-based outpatient pharmacy. Data were collected from pharmacists wearing silent, random-signal generators before and after the implementation of computer-based prescribing. MEASUREMENTS: The type of work performed by pharmacists (activity), the reason for their work (function), and the people they contacted (contact) were measured. RESULTS: Total staff hours and prescriptions handled were similar before and after computer-based prescribing. Pharmacists recorded 4,687 observations before and 4,735 observations after implementation of computer-based outpatient prescription writing. After implementation, pharmacists spent 12.9 percent more time correcting prescription problems, had 3.9 percent less idle time, and spent 2.2 percent less time in discussions with others. Pharmacists also spent 34.0 percent less time filling prescriptions, 45.8 percent more time in problem-solving activities involving prescriptions, and 3.4 percent less time providing advice. Over 80 percent of pharmacist time was spent working alone both before and after computer-based outpatient prescription writing. CONCLUSION: Computer-based prescribing results in major changes in the type of work done by hospital-based outpatient pharmacists and in the reason for their work and small changes in the people contacted during their work.

OBJECTIVE: Errors of omission are a common cause of systems failures. Physicians often fail to order tests or treatments needed to monitor/ameliorate the effects of other tests or treatments. The authors hypothesized that automated, guideline-based reminders to physicians, provided as they wrote orders, could reduce these omissions. DESIGN: The study was performed on the inpatient general medicine ward of a public teaching hospital. Faculty and housestaff from the Indiana University School of Medicine, who used computer workstations to write orders, were randomized to intervention and control groups. As intervention physicians wrote orders for 1 of 87 selected tests or treatments, the computer suggested corollary orders needed to detect or ameliorate adverse reactions to the trigger orders. The physicians could accept or reject these suggestions. RESULTS: During the 6-month trial, reminders about corollary orders were presented to 48 intervention physicians and withheld from 41 control physicians. Intervention physicians ordered the suggested corollary orders in 46.3% of instances when they received a reminder, compared with 21.9% compliance by control physicians (p < 0.0001). Physicians discriminated in their acceptance of suggested orders, readily accepting some while rejecting others. There were one third fewer interventions initiated by pharmacists with physicians in the intervention than control groups. CONCLUSION: This study demonstrates that physician workstations, linked to a comprehensive electronic medical record, can be an efficient means for decreasing errors of omissions and improving adherence to practice guidelines.

Objective: To use routine data from a comprehensive electronic medical record system to predict death among patients with reactive airways disease. Design: Retrospective cohort study conducted in an academic primary care internal medicine practice. Subjects were 1,536 adults with reactive airways disease: 542 with asthma and 994 with chronic obstructive pulmonary disease (COPD). Measurements: The dependent variable was death from any cause within 3 years following patients' first primary care appointment in 1992. Multivariable logistic regression was used to identify independent predictors of 3-year mortality, with half of the patients used to derive the predictive model and the other half used to assess its predictability. Results: Of the 1,536 study patients, 191 (12%) died in the 3-year follow-up period. From information available on or before patients' first primary care visit in 1992, multivariable predictors of 3-year mortality were coincidental heart failure, male sex, presence of COPD, lower weight, low serum albumin concentration level, and a prior arterial PO2 of less than 60 mmHg; use of an inhaled corticosteroid was protective. The c-statistic (ROC curve area) in the validation cohort was 0.76, indicating good discrimination, and goodness of fit was excellent by Hosmer-Lemeshow chi-square (P > 0.5). Only 24% of the patients in the validation cohort were designated at high risk (estimated ≥15% 3-year mortality), but this group contained more than half of the deaths within 3 years for the entire cohort. Conclusions: Data generated during routine care and stored in a comprehensive electronic medical record can accurately predict mortality among patients with reactive airways disease. Such technology can be used by practices to control for severity of illness when assessing clinical practice and to identify high-risk patients for interventions to improve prognosis.

William M. Tierney, MD, Clement J. McDonald, MD; Testing Informatics Innovations: The Value of Negative Trials, Journal of the American Medical Informatics Asso

Increasing amounts of medical knowledge, clinical data, and patient expectations have created a fertile environment for developing and using clinical practice guidelines. Electronic medical records have provided an opportunity to invoke guidelines during the everyday practice of clinical medicine to improve health care quality and control costs. In this paper, efforts to incorporate complex guidelines [those for heart failure from the Agency for Health Care Policy and Research (AHCPR)] into a network of physicians' interactive microcomputer workstations are reported. The task proved difficult because the guidelines often lack explicit definitions (e.g., for symptom severity and adverse events) that are necessary to navigate the AHCPR algorithm. They also focus more on errors of omission (not doing the right thing) than on errors of commission (doing the wrong thing) and do not account for comorbid conditions, concurrent drug therapy, or the timing of most interventions and follow-up. As they stand, the heart failure guidelines give good general guidance to individual practitioners, but cannot be used to assess quality or care without extensive "translation" into the local environment. Specific recommendations are made so that future guidelines will prove useful to a wide range of prospective users.

William M. Tierney, MD, J. Mark Overhage, MD, PhD, Clement J. McDonald, MD; A Plea for Controlled Trials in Medical Informatics, Journal of the American Medical