OBJECTIVE: To develop a framework that models the impact of electronic health record (EHR) systems on healthcare professionals' well-being and their relationships with patients, using interdisciplinary insights to guide machine learning in identifying value patterns important to healthcare professionals in EHR systems. MATERIALS AND METHODS: A theoretical framework of EHR systems' implementation was developed using interdisciplinary literature from healthcare, information systems, and management science focusing on the systems approach, clinical decision-making, and interface terminologies. OBSERVATIONS: Healthcare professionals balance personal norms of narrative and data-driven communication in knowledge creation for EHRs by integrating detailed patient stories with structured data. This integration forms 2 learning loops that create tension in the healthcare professional-patient relationship, shaping how healthcare professionals apply their values in care delivery. The manifestation of this value tension in EHRs directly affects the well-being of healthcare professionals. DISCUSSION: Understanding the value tension learning loop between structured data and narrative forms lays the groundwork for future studies of how healthcare professionals use EHRs to deliver care, emphasizing their well-being and patient relationships through a sociotechnical lens. CONCLUSION: EHR systems can improve the healthcare professional-patient relationship and healthcare professional well-being by integrating norms and values into pattern recognition of narrative and data communication forms.

The 21st Century Cures Act mandates immediate availability of test results upon request. The Cures Act does not require that patients be informed of results, but many organizations send notifications when results become available. Our medical center implemented 2 sequential policies: immediate notifications for all results, and notifications only to patients who opt in. We used over 2 years of data from Vanderbilt University Medical Center to measure the effect of these policies on rates of patient-before-clinician result review and patient-initiated messaging using interrupted time series analysis. When releasing test results with immediate notification, the proportion of patient-before-clinician review increased 4-fold and the proportion of patients who sent messages rose 3%. After transition to opt-in notifications, patient-before-clinician review decreased 2.4% and patient-initiated messaging decreased 0.4%. Replacing automated notifications with an opt-in policy provides patients flexibility to indicate their preferences but may not substantially alleviate clinicians' messaging workload.

OBJECTIVE: To improve problem list documentation and care quality. MATERIALS AND METHODS: We developed algorithms to infer clinical problems a patient has that are not recorded on the coded problem list using structured data in the electronic health record (EHR) for 12 clinically significant heart, lung, and blood diseases. We also developed a clinical decision support (CDS) intervention which suggests adding missing problems to the problem list. We evaluated the intervention at 4 diverse healthcare systems using 3 different EHRs in a randomized trial using 3 predetermined outcome measures: alert acceptance, problem addition, and National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) clinical quality measures. RESULTS: There were 288 832 opportunities to add a problem in the intervention arm and the problem was added 63 777 times (acceptance rate 22.1%). The intervention arm had 4.6 times as many problems added as the control arm. There were no significant differences in any of the clinical quality measures. DISCUSSION: The CDS intervention was highly effective at improving problem list completeness. However, the improvement in problem list utilization was not associated with improvement in the quality measures. The lack of effect on quality measures suggests that problem list documentation is not directly associated with improvements in quality measured by National Committee for Quality Assurance Healthcare Effectiveness Data and Information Set (NCQA HEDIS) quality measures. However, improved problem list accuracy has other benefits, including clinical care, patient comprehension of health conditions, accurate CDS and population health, and for research. CONCLUSION: An EHR-embedded CDS intervention was effective at improving problem list completeness but was not associated with improvement in quality measures.

Burnout among healthcare providers has been increasingly recognized as a significant problem.1 The National Academy of Medicine has defined burnout as “a syndrome characterized by high emotional exhaustion, high depersonalization (ie, cynicism), and a low sense of personal accomplishment from work.”2 The Agency for Healthcare Research and Quality similarly defines Burnout as a long-term stress reaction marked by emotional exhaustion, depersonalization, and a lack of sense of personal accomplishment.3 Clinician burnout is both costly and has been associated with reduced job satisfaction, quality and safety of care, and patient health outcomes.4 Burnout is common—affecting between 35% and 54% of U.S. nurses and physicians and between 45% and 60% for medical students and residents.2 Research to date has identified a number of contributing factors as associated with burnout.5 Among these, health information technologies (HITs) are often implicated. Electronic health record (EHR) systems, for example, are often seen as cumbersome to use, failing to fulfill the promise of improved healthcare delivery, and little more than a means of meeting regulatory and billing requirements.6 However, there remains considerable debate in the informatics community as to the actual role health information technologies play in the problem of clinician burnout.7 Existing research suggests that technologies may be confounded with other important causes, including regulatory mandates, clinical volumes, increasing hyperspecialization among healthcare providers, and a mismatch between the incentives driving system designers and purchasers and those driving providers. Regardless of the role information technology plays in clinician burnout, innovative solutions to prevent or mitigate burnout are urgently needed. In this special focus issue of Journal of the American Medical Informatics Association, we target articles evaluating the role that health information technologies have in causing and mitigating burnout, identify confounding factors, and consider informatics and policy-based solutions. This special focus issue is an outgrowth of the 2020 American College of Medical Informatics Symposium, “Clinician Burnout: Is it Informatics’ Fault, and What Can We Do About It?” In parallel with this special issue, the American College of Medical Informatics (ACMI) Symposium also led to the 25x5 initiative,8 to reduce the burden imposed by clinical documentation on healthcare providers in the United States to 25% of its current level within 5 years. The 25x5 initiative in turn resulted in a National Library of Medicine–funded 6-week symposium that concluded in February 2021. Follow-on work will lay out concrete steps that can be taken to reduce burden, create a community of like-minded stakeholders, and will work with key organizations and associations to guide this change. Taken together, the 2020 ACMI symposium, the 25x5 initiative, and this special issue comprise concrete steps the informatics community is taking to address the problem of clinician burnout. This special issue includes 24 articles across the variety of JAMIA formats: Research and Applications (n = 6),9–14 Brief Communications (n = 4),7,15–17 Reviews (n = 5),18–22 and Perspectives (n = 9).23–31 In the following paragraphs, we summarize selected papers reflecting 3 key themes: (1) understanding the relationship between HIT and clinician burnout, (2) emerging HIT approaches to mitigate clinician burnout, and (3) future directions. Several articles in this special focus issue anchor our understanding of the relationship between HIT and clinician burnout. Two review articles, one by Yan et al21 and another by Nguyen et al19 both identified documentation burden, high inbox message volumes, and negative perceptions of EHR functionality and usability as key EHR-related factors most consistently associated with objective measures of provider burnout in the extant literature. Both review articles also identified time spent on EHR after work hours—often called “pajama time”—but not total time spent on EHR, as being associated with burnout. Findings from these review articles point to opportunities for HIT systems to identify clinicians at increased risk of burnout. Baxter et al16 demonstrated that 3 leading EHR vendors currently provide “off-the-shelf” metrics to measure provider activity on the EHR through log data. While further work is needed to harmonize the metrics’ definitions so that meaningful cross-vendor comparisons can be made, these measures are now routinely available to healthcare organizations and informatics researchers. Two articles in this special issue offer practical insights on how these metrics could be used to identify the subset of clinicians at elevated risk of burnout and to target burnout mitigation interventions. Eschenroeder et al15 analyzed data from the KLAS Arch Collaborative and found that physicians who spend 6 or more hours per week performing after-hours charting were more likely to report burnout. Similarly, Peccoralo et al14 found using survey data from a single institution that faculty members who used EHR for more than 90 minutes a day after hours or who spent more than 60 minutes a day performing clerical tasks were more likely to report burnout. Taken together, these findings suggest that risk of burnout for full-time clinicians may rise significantly if they spend more than 60 to 90 minutes per day on the EHR after hours. Articles in this special issue also highlight opportunities to leverage HIT to address the pervasive problem of clinician burnout. Several contributions build on the evidence base for approaches that healthcare organizations could adopt. Lourie et al12 reported that personalized customization and training sessions across 14 specialty and 31 primary care ambulatory care practices led to improved self-reported efficiency and burnout perception. Simpson et al17 found that a 2-week EHR optimization sprint consisting of EHR changes and one-on-one training sessions led to an improvement in clinicians’ satisfaction toward the EHR in a single-specialty practice but did not impact measures of emotional burnout. A qualitative study conducted by Tran et al13 found that medical scribes are commonly used to offload 7 categories of clinical or clerical tasks as a way to a alleviate burnout attributable to the use of HIT. While these approaches require dedicated resources, these articles should help healthcare organizations build the case for these investments. As suboptimal EHR usability has often been cited as a significant contributor to clinician burnout in the United States, the editors of this special focus issue invited key EHR vendors and usability experts to elucidate current approaches to and opportunities for vendors to improve EHR usability. Leading EHR vendors were invited to respond to a semi-structured written survey on how they meet or exceed the 2015 EHR usability (or user-centered design) requirements issued by the ONC (https://www.healthit.gov/test-method/safety-enhanced-design#ccg). Anonymized responses from 4 major vendors (Supplementary Appendix) were sent to usability experts to comment on the strengths and weaknesses adopted by the EHR industry, and to suggest improvement opportunities. When compared to research from 2015 on the usability of EHR products, Hettinger et al23 noted that vendors have increased their adoption and maturity of user-centered design practices. This observation highlighted the ongoing efforts from U.S. federal policy makers, as summarized by Gettinger et al25 to promote HIT usability by implementing usability standards and funding research to examine the efficacy of these policies. However, much work remains. Hettinger et al highlighted the usability reality gap between EHR as designed by vendors and EHR as implemented by each healthcare organization, citing the paucity of the workforce trained to optimally configure and usability or safety test local configurations as a key driver of this gap. Carayon and Salwei24 further pointed out that the path for reducing clinician burnout through improving EHR usability requires a continuous approach, as vendors and their clients need to work together to turn their focus away from technology embedded in work-as-imagined toward sociotechnical systems supporting work-as-done. EHR vendors should also recognize that they can and should partner with informatics innovators to advance EHR usability and mitigate clinician burnout. In a block-randomized study, Semanik et al11 found that problem-oriented summaries of clinical data, built directly into a vendor EHR, allowed clinicians across three academic medical centers to retrieve data faster and with fewer errors. With the use of this tool, clinicians also reported a reduced cognitive load and increased satisfaction. This article by Semanik et al demonstrates how EHR vendors could support efforts to mitigate burnout by spreading and sustaining usability innovations coming from an individual customer across their customer base. So where does the topic of informatics and clinician burnout go from here? Perspectives articles from several ACMI members offer new lenses through which to view, understand, and address HIT-associated clinician burnout. Williams30 contended that moral injury associated with EHR, as defined by EHR use that leads clinicians to transgress deeply held moral beliefs and expectations, may be a hidden contributor to clinician burnout. Weir et al31 further postulated that burnout may be linked to drivers of intrinsic motivation, and that goal-based decision making, sense making, and agency or autonomy should be considered in the design of future technological interventions to mitigate clinician burnout. From a methodological perspective, significant opportunities remain. Moy et al20 pointed out in their scoping review that standard and validated measures of documentation burden are still lacking, which in turn forms a barrier to the rigorous study of documentation burden. Moy et al further called for efforts to operationalize the concept of documentation burden and develop best practices for measurement. Kannampallil et al26 proposed a conceptual framework that would allow the informatics community to build on EHR activity measures evaluated by Baxter et al16 and use technology to assess holistically clinicians’ workload, cognitive burden, and well-being. The editors of this JAMIA special issue recognize that this body of work is but a snapshot of a rapidly growing and evolving topic. New technologies such as ambient voice speech to text, internet of things, natural language processing and machine learning–driven data visualization, and Fast Healthcare Interoperability Resources, as highlighted by Dymek et al28 and Gettinger and Zayas-Cabán,25 may yet open up opportunities to support more meaningful clinician-patient interactions and more efficient workflows. The policy landscape is also constantly changing, as evidenced by the recent simplification in documentation requirements initiated by the Centers for Medicare and Medicaid Services Burden Reduction efforts intended to place “Patients Over Paperwork.”32 As a target for multidisciplinary scientific inquiry, the subject of HIT-associated clinician burnout must continue to evolve through future empirical studies. Its current evidence base remains modest, at best. We therefore encourage readers of this special issue to participate in and accelerate the ongoing work in this area. STR, KZ, and EGP presented the special issue proposal to JAMIA; STR, KZ, and EGP fulfilled Associate Editor duties; EGP fulfilled EIC duties; EGP contributed to instrument design, data collection from vendors and commissioning of invited perspectives articles; and EGP, STR, and KZ contributed to drafting and finalization of the editorial. Supplementary material is available at Journal of the American Medical Informatics Association online. The authors have no relevant conflicts of interest to declare.

The COVID-19 national emergency has led to surging care demand and the need for unprecedented telehealth expansion. Rapid telehealth expansion can be especially complex for pediatric patients. From the experience of a large academic medical center, this report describes a pathway for efficiently increasing capacity of remote pediatric enrollment for telehealth while fulfilling privacy, security, and convenience concerns. The design and implementation of the process took 2 days. Five process requirements were identified: efficient enrollment, remote ability to establish parentage, minimal additional work for application processing, compliance with guidelines for adolescent autonomy, and compliance with institutional privacy and security policies. Weekly enrollment subsequently increased 10-fold for children (age 0-12 years) and 1.2-fold for adolescents (age 13-17 years). Weekly telehealth visits increased 200-fold for children and 90-fold for adolescents. The obstacles and solutions presented in this report can provide guidance to health systems for similar challenges during the COVID-19 response and future disasters.

Emergent policy changes related to telemedicine and the Emergency Medical Treatment and Labor Act during the novel coronavirus disease 2019 (COVID-19) pandemic have created opportunities for technology-based clinical evaluation, which serves to conserve personal protective equipment (PPE) and protect emergency providers. We define electronic PPE as an approach using telemedicine tools to perform electronic medical screening exams while satisfying the Emergency Medical Treatment and Labor Act. We discuss the safety, legal, and technical factors necessary for implementing such a pathway. This approach has the potential to conserve PPE and protect providers while maintaining safe standards for medical screening exams in the emergency department for low-risk patients in whom COVID-19 is suspected.

With advances in technology, patients increasingly expect to access their health information on their phones and computers seamlessly, whenever needed, to meet their clinical needs. The 1996 passage of the Health Insurance Portability and Accountability Act (HIPAA), modifications made by the Health Information Technology for Economic and Clinical Health Act (HITECH), and the recent 21st Century Cures Act (Cures) promise to make patients' health information available to them without special effort and at no cost. However, inconsistencies among these policies' definitions of what is included in "health information", widespread variation in electronic health record system capabilities, and differences in local health system policies around health data release have created a confusing landscape for patients, health care providers, and third parties who reuse health information. In this article, we present relevant regulatory history, describe challenges to health data portability and fluidity, and present the authors' policy recommendations for lawmakers to consider so that the vision of HIPAA, HITECH, and Cures may be fulfilled.

OBJECTIVE: Phenotyping algorithms applied to electronic health record (EHR) data enable investigators to identify large cohorts for clinical and genomic research. Algorithm development is often iterative, depends on fallible investigator intuition, and is time- and labor-intensive. We developed and evaluated 4 types of phenotyping algorithms and categories of EHR information to identify hypertensive individuals and controls and provide a portable module for implementation at other sites. MATERIALS AND METHODS: We reviewed the EHRs of 631 individuals followed at Vanderbilt for hypertension status. We developed features and phenotyping algorithms of increasing complexity. Input categories included International Classification of Diseases, Ninth Revision (ICD9) codes, medications, vital signs, narrative-text search results, and Unified Medical Language System (UMLS) concepts extracted using natural language processing (NLP). We developed a module and tested portability by replicating 10 of the best-performing algorithms at the Marshfield Clinic. RESULTS: Random forests using billing codes, medications, vitals, and concepts had the best performance with a median area under the receiver operator characteristic curve (AUC) of 0.976. Normalized sums of all 4 categories also performed well (0.959 AUC). The best non-NLP algorithm combined normalized ICD9 codes, medications, and blood pressure readings with a median AUC of 0.948. Blood pressure cutoffs or ICD9 code counts alone had AUCs of 0.854 and 0.908, respectively. Marshfield Clinic results were similar. CONCLUSION: This work shows that billing codes or blood pressure readings alone yield good hypertension classification performance. However, even simple combinations of input categories improve performance. The most complex algorithms classified hypertension with excellent recall and precision.

OBJECTIVE: The goal of this study was to develop a practical framework for recognizing and disambiguating clinical abbreviations, thereby improving current clinical natural language processing (NLP) systems' capability to handle abbreviations in clinical narratives. METHODS: We developed an open-source framework for clinical abbreviation recognition and disambiguation (CARD) that leverages our previously developed methods, including: (1) machine learning based approaches to recognize abbreviations from a clinical corpus, (2) clustering-based semiautomated methods to generate possible senses of abbreviations, and (3) profile-based word sense disambiguation methods for clinical abbreviations. We applied CARD to clinical corpora from Vanderbilt University Medical Center (VUMC) and generated 2 comprehensive sense inventories for abbreviations in discharge summaries and clinic visit notes. Furthermore, we developed a wrapper that integrates CARD with MetaMap, a widely used general clinical NLP system. RESULTS AND CONCLUSION: CARD detected 27 317 and 107 303 distinct abbreviations from discharge summaries and clinic visit notes, respectively. Two sense inventories were constructed for the 1000 most frequent abbreviations in these 2 corpora. Using the sense inventories created from discharge summaries, CARD achieved an F1 score of 0.755 for identifying and disambiguating all abbreviations in a corpus from the VUMC discharge summaries, which is superior to MetaMap and Apache's clinical Text Analysis Knowledge Extraction System (cTAKES). Using additional external corpora, we also demonstrated that the MetaMap-CARD wrapper improved MetaMap's performance in recognizing disorder entities in clinical notes. The CARD framework, 2 sense inventories, and the wrapper for MetaMap are publicly available at https://sbmi.uth.edu/ccb/resources/abbreviation.htm . We believe the CARD framework can be a valuable resource for improving abbreviation identification in clinical NLP systems.

People are increasingly capturing health, wellness, and clinical data about themselves using a growing palette of inexpensive and pervasive technologies. These technologies allow people to record, analyze, and curate health data outside of settings where health care is traditionally delivered, and without consistently involving health care professionals. Examples include wrist-worn accelerometers with software that calculates daily footsteps and sleep, global positioning service–enabled devices that track miles run or biked, Web-based health journaling tools, smart online food diaries, and networked weight scales or blood pressure machines. People also use online resources such as social networks to help them use and interpret data obtained by these technologies as well as data originating from more traditional medical testing. In many cases, these technologies can complement, or even replace, interactions with health care professionals. Recognizing that people often use these technologies independent of situations where they are patients per se (eg, when navigating the health care system or as part of a prescribed treatment program) we term them “person-generated health data” (PGHD) technologies. This JAMIA special issue focuses on technologies people use to record, manage, interpret, and display data representing health, wellness, and clinically related activities. Interpreting the importance of data produced by PGHD technologies for individual people can be challenging. The growing availability and widespread adoption of PGHD technologies lead to a number of key informatics issues. Currently, the field lacks a clear theoretical basis, a set of data models, and empirically derived strategies for integrating tools and data into existing clinical applications and workflows. Research focused on evaluating PGHD and its utility in health care decision-making has not kept up with the rapidly growing market for these technologies. This special issue was assembled to provide a forum for investigators conducting research or conceptualizing perspectives on PGHD to present their work. Manuscripts in this issue highlight the relative youth of research on PGHD technologies and showcase the diversity of approaches for using PGHD as part of health care. In general, manuscripts included here present conceptual models for integrating these technologies with larger health care systems, small preliminary studies using them in specific health care settings, and approaches to gathering PGHD. Taken as a whole, these manuscripts articulate a need for further research with larger subject sample sizes and longer-term clinical outcomes. This special issue also presents a few large trials involving more traditional technologies. A number of manuscripts lay out conceptual frameworks for using PGHD technologies for clinical care and research. Chung and colleagues describe the Crohn’s and Colitis Foundation of America (CCFA) Partners Patient-Powered Research Network, funded by the Patient Centered Outcomes Research Institute (see page 485). CCFA brings together more than 14 000 patients with inflammatory bowel disease who use a number of PGHD technologies, including a social network platform called Crohnology and a cloud-based platform that connects numerous mobile health devices and wearables. This manuscript lays out the workflows, policies, methods, data, and data standards that the CCFA is leveraging as it builds out its research network. Likewise, in the domain of irritable bowel disease, Karkar, Kientz, and colleagues present a framework for using mobile applications to support individual self-experimentation as part of chronic disease self-management (see page 440). Based on a literature review and a small qualitative study, this perspective article recommends a generalizable approach to creating mobile applications that allow people with chronic disease to perform self-experimentation when managing it. This approach includes a step where patients generate a list of study questions they want to evaluate themselves, in which they record their choices and outcomes and symptoms, and a step where they can review the results. Petersen offers a perspective about using PGHD to manage cancer survivors’ care needs at the individual and population levels (see page 456). The manuscript provides models for using PGHD to: (1) manage survivors’ autonomy, (2) coordinate information among different health care and complementary medicine providers, (3) recognize changes in health status, (4) provide risk stratification across a population of survivors, and (5) support secondary prevention and screening efforts. The manuscript goes on to speculate that PGHD technologies may also help to bridge gaps with relatively underserved populations in which long-term cancer survivorship needs are often unmet. It concludes by articulating that maximizing PGHD’s potential requires providers and survivors to work together in an environment that promotes transparency, trust, and shared decision-making. Additionally, Woods and colleagues present a conceptual framework containing 10 specific recommendations from the US Department of Veterans Affairs for collecting and managing PGHD from patients and using novel mobile technologies (see page 491). Recommendations for institutions planning to incorporate PGHD include taking the time to advance organizational change, identifying shared value for patients and their health care providers, respecting data standards, and innovating in the areas of technology, visualization, and policy, among others. The special issue also includes a number of small studies reporting early findings about novel technical infrastructures for studying or connecting PGHD technologies. In a small study by Shaw and colleagues (see page 462), investigators developed and piloted a technological infrastructure designed to collect and analyze mobile health data from multiple devices and applied it to a small patient sample to measure data-collection rates over time. The infrastructure collected and combined diverse data from different platforms and software applications, such as step counts, activity intensity, sleep quantity, blood pressure, capillary blood oxygen saturation, weight, and fluid intake. Subjects were willing to use these systems, but there were trends toward lower usage among those with chronic illness and over time. In another study, Mamykina and colleagues evaluated decision-making strategies among individuals with diabetes and nutritionists when presented with images of meals (see page 526). In a controlled setting, this study simulated common app-assisted dietary self-monitoring and forecasting activities to elucidate how people reason about dietary choices. The authors observed that reasoning was complex, and that diabetes forecasting based on dietary observations and existing journaling methods could be unreliable. Using off-the-shelf mobile technologies, Kumar and colleagues addressed diabetes self-management (see page 532). Connecting widely available software tools published by well-known smartphone vendors, health information technology system vendors, and ambulatory glucose meter vendors, the investigators integrated a continuous feed of home-based glucose monitoring into an electronic medical record. The article reviews many of the issues related to workflow, visualization, and escalation that arose among the few patients who pilot-tested the system. In one of the larger studies in this special issue, Johnson and colleagues randomized adolescent subjects to receive (a) regular text message reminders about their asthma management based on self-curated reminder schedules or (b) a more traditional approach to asthma management (control group) (see page 449). The study demonstrated that subjects who received text message reminders had higher satisfaction, improved engagement and self-efficacy, and better medical adherence. However, the intervention’s effect magnitude was lower in some subgroups, with lowest use among African American subjects. Of interest, a number of studies address the use of PGHD technologies for mental health disorders. Murnane and colleagues surveyed a large number of individuals with bipolar disorder to assess the methods they had spontaneously chosen to self-monitor aspects of their disease (see page 477). Subject monitoring revealed a number of issues, including those related to mood, sleep, finances, exercise, and social interactions, through the use of digital tracking methods. Subjects also noted that digital self-monitoring tools allowed self-reflection, better health management, and improved communication with treatment teams. In another small study, Abdullah and colleagues evaluated the feasibility of automatically inferring a validated marker of stability and rhythmicity for individuals with bipolar disorder using data passively collected from their smartphones (see page 538). The study included location, distance travelled, conversation frequency, and nonstationary duration, and correlated these items with self-assessed symptom control. While this formative study was limited to just a few subjects, it demonstrated that automatic smartphone sensing might be used in the future to infer the well-being of people with bipolar disorder. Erfani and colleagues used qualitative methods to evaluate a social networking website related to promoting better psychological well-being of cancer-affected people (see page 467). In their study, 25 women with ovarian cancer who accessed the website participated in semistructured interviews. Most participants accessed the website daily, with varying levels of activity in creating new content. Subjects felt that using the social networking site improved their psychological well-being by increasing their sense of social support and social connectedness. A portion of the articles address mental health issues related to using PGHD technologies and the psychological well-being of people with chronic medical conditions. Deetjen and Powell also evaluated a health-related social networking website in terms of how people with a chronic health condition used it (see page 508). Using a Bayesian classification algorithm, the study evaluated whether social network postings were more informational or emotional in nature, based on the presence of 1 of 14 underlying health conditions. The data demonstrated that the tendency toward emotional posts differed by condition, and were most common among posts about mental health conditions from people who were younger and who posted relatively more frequently. By contrast, informational postings related more to nonterminal physical conditions, such as diabetes and asthma. Hartzler and colleagues evaluated a method for identifying potential peer mentors for people who use social networks as support for a chronic health condition (see page 496). In this mixed-methods study, a few social media users with cancer reviewed and ranked potential peer mentors who were automatically identified based on their health interests, language style, demographics, and sample posts. Among these, only sample posts predicted whether subjects subsequently contacted a potential mentor. Subjects articulated how important it is for peer mentors to be knowledgeable, sociable, and articulate. In a study by Sanger and colleagues, the authors conducted a formative evaluation and identified approaches to engage patients in postsurgical wound management using PGHD technologies (see page 514). Subjects, including patients and health care providers, identified 4 critical components for wound monitoring: (1) the software should provide contextual metadata containing patients’ clinical information and measures of adherence; (2) the data should be easily accessible and actionable in their presentation; (3) the software tools should build on existing sociotechnical systems, such as workflows and health information technologies; and (4) the process of data flow and interpretation should be fully transparent to all users. The study identified some differences among subjects, including variable flexibility in data input, response prioritization, etc. The field of PGHD and related technologies remains young and is quickly changing based on the endless arrival of new devices, applications, and interfaces. The real value these technologies add to health care delivery remains to be seen. Substantial research across a number of areas is still needed, for example, research to develop an ontology of PGHD that includes elements that may be defined by individuals or are captured through existing clinical processes. Research is also needed to evaluate the usage of technologies that capture and display PGHD, including validity, reliability, and utility to different potential stakeholders; what constitutes best practices around infrastructure, user interface, and security; and how novel models of use, such as social networks and gamification, influence the success of these technologies.

In alignment with a major shift toward patient-centered care as the model for improving care in our health system, informatics is transforming patient-provider relationships and overall care delivery. AMIA's 2013 Health Policy Invitational was focused on examining existing challenges surrounding full engagement of the patient and crafting a research agenda and policy framework encouraging the use of informatics solutions to achieve this goal. The group tackled this challenge from educational, technical, and research perspectives. Recommendations include the need for consumer education regarding rights to data access, the need for consumers to access their health information in real time, and further research on effective methods to engage patients. This paper summarizes the meeting as well as the research agenda and policy recommendations prioritized among the invited experts and stakeholders.

OBJECTIVE: The general equivalence mappings (GEMs) and reimbursement mappings (RMs) facilitate translation between ICD-9-CM and ICD-10-CM. This study compared prospectively dual-encoded diagnoses assigned by professional coders with the GEMs/RMs in a clinical setting. MATERIALS AND METHODS: Professional coders manually encoded diagnoses from 100 primary care notes into both ICD-9-CM and ICD-10-CM. The investigators evaluated whether manual mappings were reproducible using the GEMs/RMs. Reproducible mappings with one ICD-9-CM and one ICD-10-CM code ("one-to-one") were classified as exact or approximate using GEMs flags. Mismatches were characterized manually. RESULTS: Manual encodings were reproducible from the forward GEMs, backward GEMs, and RMs in 85.2%, 90.4%, and 88.1% of diagnoses, respectively. For one-to-one, reproducible mappings, 61% (forward) and 63% (backward) were approximate mappings compared to 85% and 95% in the GEMs as a whole. Mismatches between manual and GEMs encodings were due to differences in coder interpretation (11%-13%), subtle hierarchical differences (52%-55%), or unknown reasons (32%-35%). DISCUSSION: This study highlights inconsistencies between manual encoding and using the GEMs/RMs. The number of approximate mappings in our population compared to all one-to-one GEMs entries supports the notion that statistics describing the GEMs as a whole might not represent the most important mappings for each organization. The mismatch characteristics highlight the subtle differences between manual encoding and using the GEMs/RMs. CONCLUSION: These results support the need for organizations to assess the GEMs and RMs in their own environment to avoid changes in reimbursement and longitudinal statistics.

The Mid-South Clinical Data Research Network (CDRN) encompasses three large health systems: (1) Vanderbilt Health System (VU) with electronic medical records for over 2 million patients, (2) the Vanderbilt Healthcare Affiliated Network (VHAN) which currently includes over 40 hospitals, hundreds of ambulatory practices, and over 3 million patients in the Mid-South, and (3) Greenway Medical Technologies, with access to 24 million patients nationally. Initial goals of the Mid-South CDRN include: (1) expansion of our VU data network to include the VHAN and Greenway systems, (2) developing data integration/interoperability across the three systems, (3) improving our current tools for extracting clinical data, (4) optimization of tools for collection of patient-reported data, and (5) expansion of clinical decision support. By 18 months, we anticipate our CDRN will robustly support projects in comparative effectiveness research, pragmatic clinical trials, and other key research areas and have the capacity to share data and health information technology tools nationally.

The patient portal is a relatively new healthcare information technology that enables patients more convenient access to their healthcare information and allows them to send messages to their doctors. Our study examines the themes discussed in these messages and the different ways in which patients communicate with their providers via a portal employed in a large medical center. We also explore the differences between the patient portal and more traditional communication media, and investigated the advantages and potential problems of the portal system. Our findings show a wide variety of topics discussed in the communication messages (such as medication, appointments, laboratory tests, etc.) and how patients provide information, consult with their providers, and express psychosocial and emotional needs. We argue that the patient portal improves the accuracy of communication and could facilitate illness management for patients, especially over a longer term. However, messaging through the patient portal is not popular among patients and the simultaneous use of multiple communication media may create information gaps. More research is needed to better elucidate barriers to the use of patient portals and the optimal methods of communication and information integration given different contexts.

Much of what is currently documented in the electronic health record is in response toincreasingly complex and prescriptive medicolegal, reimbursement, and regulatory requirements. These requirements often result in redundant data capture and cumbersome documentation processes. AMIA's 2011 Health Policy Meeting examined key issues in this arena and envisioned changes to help move toward an ideal future state of clinical data capture and documentation. The consensus of the meeting was that, in the move to a technology-enabled healthcare environment, the main purpose of documentation should be to support patient care and improved outcomes for individuals and populations and that documentation for other purposes should be generated as a byproduct of care delivery. This paper summarizes meeting deliberations, and highlights policy recommendations and research priorities. The authors recommend development of a national strategy to review and amend public policies to better support technology-enabled data capture and documentation practices.

Vanderbilt University has a widely adopted patient portal, MyHealthAtVanderbilt, which provides an infrastructure to deliver information that can empower patient decision making and enhance personalized healthcare. An interdisciplinary team has developed Flu Tool, a decision-support application targeted to patients with influenza-like illness and designed to be integrated into a patient portal. Flu Tool enables patients to make informed decisions about the level of care they require and guides them to seek timely treatment as appropriate. A pilot version of Flu Tool was deployed for a 9-week period during the 2010-2011 influenza season. During this time, Flu Tool was accessed 4040 times, and 1017 individual patients seen in the institution were diagnosed as having influenza. This early experience with Flu Tool suggests that healthcare consumers are willing to use patient-targeted decision support. The design, implementation, and lessons learned from the pilot release of Flu Tool are described as guidance for institutions implementing decision support through a patient portal infrastructure.

OBJECTIVE: The authors' goal was to develop and evaluate machine-learning-based approaches to extracting clinical entities-including medical problems, tests, and treatments, as well as their asserted status-from hospital discharge summaries written using natural language. This project was part of the 2010 Center of Informatics for Integrating Biology and the Bedside/Veterans Affairs (VA) natural-language-processing challenge. DESIGN: The authors implemented a machine-learning-based named entity recognition system for clinical text and systematically evaluated the contributions of different types of features and ML algorithms, using a training corpus of 349 annotated notes. Based on the results from training data, the authors developed a novel hybrid clinical entity extraction system, which integrated heuristic rule-based modules with the ML-base named entity recognition module. The authors applied the hybrid system to the concept extraction and assertion classification tasks in the challenge and evaluated its performance using a test data set with 477 annotated notes. MEASUREMENTS: Standard measures including precision, recall, and F-measure were calculated using the evaluation script provided by the Center of Informatics for Integrating Biology and the Bedside/VA challenge organizers. The overall performance for all three types of clinical entities and all six types of assertions across 477 annotated notes were considered as the primary metric in the challenge. RESULTS AND DISCUSSION: Systematic evaluation on the training set showed that Conditional Random Fields outperformed Support Vector Machines, and semantic information from existing natural-language-processing systems largely improved performance, although contributions from different types of features varied. The authors' hybrid entity extraction system achieved a maximum overall F-score of 0.8391 for concept extraction (ranked second) and 0.9313 for assertion classification (ranked fourth, but not statistically different than the first three systems) on the test data set in the challenge.

Explicit guidelines are needed to develop safe and effective patient portals. This paper proposes general principles, policies, and procedures for patient portal functionality based on MyHealthAtVanderbilt (MHAV), a robust portal for Vanderbilt University Medical Center. We describe policies and procedures designed to govern popular portal functions, address common user concerns, and support adoption. We present the results of our approach as overall and function-specific usage data. Five years after implementation, MHAV has over 129,800 users; 45% have used bi-directional messaging; 52% have viewed test results and 45% have viewed other medical record data; 30% have accessed health education materials; 39% have scheduled appointments; and 29% have managed a medical bill. Our policies and procedures have supported widespread adoption and use of MHAV. We believe other healthcare organizations could employ our general guidelines and lessons learned to facilitate portal implementation and usage.

Clinical documentation is central to patient care. The success of electronic health record system adoption may depend on how well such systems support clinical documentation. A major goal of integrating clinical documentation into electronic heath record systems is to generate reusable data. As a result, there has been an emphasis on deploying computer-based documentation systems that prioritize direct structured documentation. Research has demonstrated that healthcare providers value different factors when writing clinical notes, such as narrative expressivity, amenability to the existing workflow, and usability. The authors explore the tension between expressivity and structured clinical documentation, review methods for obtaining reusable data from clinical notes, and recommend that healthcare providers be able to choose how to document patient care based on workflow and note content needs. When reusable data are needed from notes, providers can use structured documentation or rely on post-hoc text processing to produce structured data, as appropriate.

OBJECTIVES: Improvements in electronic health record (EHR) system development will require an understanding of psychiatric clinicians' views on EHR system acceptability, including effects on psychotherapy communications, data-recording behaviors, data accessibility versus security and privacy, data quality and clarity, communications with medical colleagues, and stigma. DESIGN: Multidisciplinary development of a survey instrument targeting psychiatric clinicians who recently switched to EHR system use, focus group testing, data analysis, and data reliability testing. MEASUREMENTS: Survey of 120 university-based, outpatient mental health clinicians, with 56 (47%) responding, conducted 18 months after transition from a paper to an EHR system. RESULTS: Factor analysis gave nine item groupings that overlapped strongly with five a priori domains. Respondents both praised and criticized the EHR system. A strong majority (81%) felt that open therapeutic communications were preserved. Regarding data quality, content, and privacy, clinicians (63%) were less willing to record highly confidential information and disagreed (83%) with including their own psychiatric records among routinely accessed EHR systems. LIMITATIONS: single time point; single academic medical center clinic setting; modest sample size; lack of prior instrument validation; survey conducted in 2005. CONCLUSIONS: In an academic medical center clinic, the presence of electronic records was not seen as a dramatic impediment to therapeutic communications. Concerns regarding privacy and data security were significant, and may contribute to reluctances to adopt electronic records in other settings. Further study of clinicians' views and use patterns may be helpful in guiding development and deployment of electronic records systems.

OBJECTIVE: Interface terminologies are designed to support interactions between humans and structured medical information. In particular, many interface terminologies have been developed for structured computer based documentation systems. Experts and policy-makers have recommended that interface terminologies be mapped to reference terminologies. The goal of the current study was to evaluate how well the reference terminology SNOMED CT could map to and represent two interface terminologies, MEDCIN and the Categorical Health Information Structured Lexicon (CHISL). DESIGN: Automated mappings between SNOMED CT and 500 terms from each of the two interface terminologies were evaluated by human reviewers, who also searched SNOMED CT to identify better mappings when this was judged to be necessary. Reviewers judged whether they believed the interface terms to be clinically appropriate, whether the terms were covered by SNOMED CT concepts and whether the terms' implied semantic structure could be represented by SNOMED CT. MEASUREMENTS: Outcomes included concept coverage by SNOMED CT for study terms and their implied semantics. Agreement statistics and compositionality measures were calculated. RESULTS: The SNOMED CT terminology contained concepts to represent 92.4% of MEDCIN and 95.9% of CHISL terms. Semantic structures implied by study terms were less well covered, with some complex compositional expressions requiring semantics not present in SNOMED CT. Among sampled terms, those from MEDCIN were more complex than those from CHISL, containing an average 3.8 versus 1.8 atomic concepts respectively, p<0.001. CONCLUSION: Our findings support using SNOMED CT to provide standardized representations of information created using these two terminologies, but suggest that enriching SNOMED CT semantics would improve representation of the external terms.

Preventive care measures remain underutilized despite recommendations to increase their use. The objective of this review was to examine the characteristics, types, and effects of paper- and computer-based interventions for preventive care measures. The study provides an update to a previous systematic review. We included randomized controlled trials that implemented a physician reminder and measured the effects on the frequency of providing preventive care. Of the 1,535 articles identified, 28 met inclusion criteria and were combined with the 33 studies from the previous review. The studies involved 264 preventive care interventions, 4,638 clinicians and 144,605 patients. Implementation strategies included combined paper-based with computer generated reminders in 34 studies (56%), paper-based reminders in 19 studies (31%), and fully computerized reminders in 8 studies (13%). The average increase for the three strategies in delivering preventive care measures ranged between 12% and 14%. Cardiac care and smoking cessation reminders were most effective. Computer-generated prompts were the most commonly implemented reminders. Clinician reminders are a successful approach for increasing the rates of delivering preventive care; however, their effectiveness remains modest. Despite increased implementation of electronic health records, randomized controlled trials evaluating computerized reminder systems are infrequent.

BACKGROUND: The authors define a DNA biobank as a repository of genetic information correlated with patient medical records. DNA biobanks may assist in the research and identification of genetic factors influencing disease and drug interactions, but may raise ethical issues. How healthcare providers perceive DNA biobanks is unknown. OBJECTIVES: To determine how useful healthcare professionals believe DNA biobanks will be and whether these attitudes differ between private and socialized healthcare systems. DESIGN: The authors surveyed 200 healthcare professionals, including research and non-research focused doctors, nurses and other staff from medical centers and independent practice in both the United States and Scotland. The survey included fifteen items evaluated for general receptiveness toward biobanks, presumed usefulness of biobanks and perceived attitudes in recruiting patients for a biobank. MEASUREMENTS: A total of 81 (45%) of 179 eligible participants responded: 41 from the U.S. and 40 from Scotland. Of these respondents, most (70%) were from academic centers. RESULTS: Results indicate that there is a broadly favorable attitude in both locations toward the creation of a DNA biobank (83%) and its perceived benefit (75%). This enthusiasm is tempered in Scotland when respondents evaluated their comfort in consenting patients for entry into a biobank; 16 of 40 respondents (40%) were uncomfortable doing so, representing a significant difference from those in the U.S. (p=0.001). CONCLUSIONS: Despite systematic differences in healthcare practice between the U.S. and Scotland, health care professionals in both nations believe DNA biobanks will be useful in curing disease. This finding appears to support further development of such a research tool.

OBJECTIVE: Evaluations of individual terminology systems should be driven in part by the intended usages of such systems. Clinical interface terminologies support interactions between healthcare providers and computer-based applications. They aid practitioners in converting clinical "free text" thoughts into the structured, formal data representations used internally by application programs. Interface terminologies also serve the important role of presenting existing stored, encoded data to end users in human-understandable and actionable formats. The authors present a model for evaluating functional utility of interface terminologies based on these intended uses. DESIGN: Specific parameters defined in the manuscript comprise the metrics for the evaluation model. MEASUREMENTS: Parameters include concept accuracy, term expressivity, degree of semantic consistency for term construction and selection, adequacy of assertional knowledge supporting concepts, degree of complexity of pre-coordinated concepts, and the "human readability" of the terminology. The fundamental metric is how well the interface terminology performs in supporting correct, complete, and efficient data encoding or review by humans. RESULTS: Authors provide examples demonstrating performance of the proposed evaluation model in selected instances. CONCLUSION: A formal evaluation model will permit investigators to evaluate interface terminologies using a consistent and principled approach. Terminology developers and evaluators can apply the proposed model to identify areas for improving interface terminologies.

Both researchers and policy-making organizations have identified electronic prescribing (e-prescribing) systems as a means to improve quality in healthcare delivery. However, there are documented risks involved in integrating such clinical systems into healthcare processes.1–6 These risks generally fall into two categories. First, new technologies may not accomplish what they are designed to do. Second, introduction of new technologies may lead to unintended consequences such as patient harm or misused resources. To determine whether they work as expected and without incurring risk, people and institutions implementing e-prescribing should evaluate their systems in the context of pre-existing processes. The current issue of the Journal of the American Medical Informatics Association (JAMIA) includes five articles that evaluate e-prescribing tools or related clinical workflows. These papers include a detailed prescription workflow analysis, a study of the diffusion of drug withdrawal knowledge to reference texts, and three studies that evaluate the impact of various forms of medication-related decision support, including adverse drug event monitoring. Taken together, the JAMIA papers illustrate a variety of methodological approaches to evaluation, and demonstrate some of the challenges related to evaluating e-prescribing. The major reason to evaluate e-prescribing systems is to determine how their use improves or impairs clinical and process-related outcomes. Evaluators of such systems have a palette of study methodologies from which to choose.7 For example, they may simply observe and describe past or current prescribing conditions. Alternatively, they may introduce a prospective intervention into a clinical environment and measure the impact of the change. Investigators generally select a study design methodology based on its ability and relevance to demonstrating associations between one or more factors and an outcome of interest. For example, one design may be preferred over another based on how well it can validate reduced medication error rates for e-prescribing systems. Randomized, prospective controlled trials are often considered the most definitive methodology for demonstrating such associations. By contrast, individual case reports documenting observations from one site may do little to convince skeptics that an association exists. Prospective controlled trials allow investigators to distribute subjects into two or more study groups and to control, artificially, a single factor, varying it from group to group. Investigators then measure differences among the groups and draw conclusions based on observed similarities or differences. In case reports, investigators describe in retrospect a single observation (“case”) that “occurred in the real world” and then speculate about general lessons learned from the case. There also exist hybrid, or “quasi-experimental”, methods for evaluating observed events.7 Such methods, which include case-controlled studies and time series analyses, permit investigators to apply robust statistical algorithms to study the impact of naturally occurring changes or events on populations. Prospective controlled study designs, while ideal for showing associations and causality, are challenging to implement in clinical informatics. The e-prescribing-related manuscripts in the current issue of JAMIA illustrate three important challenges to conducting such studies. Investigators should strive 1) to understand and articulate the nature of the intervention as actually implemented; 2) to define the most appropriate unit of study and analysis; and, 3) to randomize subjects in a manner that takes workflow considerations into account. The validity and generalizability of informatics evaluations result from careful attention paid to selecting and implementing appropriate analytical methods. This discussion will use the term “factor” to refer to any attribute of an e-prescribing system, of a workflow, or of an individual subject that might reasonably influence an observed outcome. The first challenge in evaluating e-prescribing (and other clinical informatics) systems involves identification of a specific factor that differs among study groups, and whose variation is expected to correlate with important study outcomes—i.e., whether measurable data can support a relationship between an isolated factor and a measurable outcome. To establish that observed differences are solely due to the individual factor under study, investigators must ensure that the only difference among study groups is the factor in question. However, in clinical settings where the intervention is an informatics (e.g., e-prescribing) system, isolating the effect of one specific factor in the study from others in the environment can be challenging. Unmeasured systematic differences between study groups may cause observed differences in outcomes. For example, in the current issue of JAMIA, McGregor and colleagues evaluated whether a decision support tool could reduce inappropriate antibiotic prescriptions in a hospital.8 In the study, an antimicrobial management team interacted with an investigational decision support system designed to alert team members when an antibiotic order was potentially inappropriate. As study subjects, antimicrobial management team members were exposed both to decision support alerts and to a change in their workflows that included time for focused chart review. While it is possible that the decision support alerts per se led to the observed reduction in antibiotic costs and total team workload, it is also possible that workflow changes necessary to deliver the decision support could have contributed to these outcomes. A second challenge during informatics systems evaluations is to identify the most appropriate individual or entity to serve as the “unit of study,” or the “study subject,” that is exposed to or influenced by the factor under study. Study subjects may consist of individual patients, single healthcare providers, complete hospital wards or entire hospitals. For example, investigators studying a new e-prescribing system's impact on error rates might calculate ward-specific error rates in a hospital where the system was present on five wards and not present on ten other wards; in this case, each hospital ward would comprise a single “study subject.” Investigators should determine statistical power, acquire results data, and compare outcomes as they directly involve the study subject. In the current issue of JAMIA, the study reported by Kilbridge and colleagues compared the rates of adverse drug events reported by automated and manual systems both at an academic hospital and at a community hospital.9 The authors hypothesized that having an academic affiliation would impact the hospital's adverse drug event rates. The entire hospital was subjected to the exposure under study, specifically, whether it was an academic teaching hospital or not. Therefore, the most appropriate unit of study (i.e., study subject) would have been the entire hospital. The authors elected instead to compare per-patient adverse drug event rates. While this approach may have been valid, other hospital-wide differences between the two sites, such as admission rates, available subspecialty services, workflow variation, and various unmeasured factors might have caused the observed differences. A third challenge to informatics systems investigators is to minimize the chance that subjects will cross over among study conditions when randomized into study groups. Investigators randomly assign subjects to study groups with the goal of increasing the likelihood that each group has a uniform composition before being subjected to the study conditions. Random assignment accomplishes this goal by evenly distributing among study groups all factors that may impact the outcome under investigation. For example, random assignment in a study evaluating the impact of e-prescribing on prescribing errors would be expected to create several study groups, each with an equivalent number of subjects who are comfortable using computers, who represent various clinical roles, and whose ages fall within similar ranges. Investigators typically randomize based on the unit of study, as defined above. As Johnson and Fitzhenry describe in the current issue of JAMIA,10 processes surrounding e-prescribing workflows involve many people who have different clinical roles. The presence of complex workflows can increase the chance that an individual study subject will crossover from one study group to others, and thus be exposed to different experimental conditions. Crossing over can reduce differences in observed results among study groups. To mediate against this risk, studies of e-prescribing systems should attempt to randomize subjects by defining the unit of study based on workflow considerations. The current manuscript by Judge and colleagues provides a good example of how this can be done. In that study, three intact, self-contained long-term care facility patient units were randomly assigned to receive decision support messages during order entry, while four control units received no such messages.11 Randomizing by entire ward likely decreased the risk that individual study subjects crossed from an intervention group into a control group, or vice versa. For cases where certain care-team members work in multiple units, it may make more sense to randomize using even larger blocks, such as entire facilities. Designing and conducting prospective comparative and controlled studies should be a goal for all informatics evaluators. However, this is not always easily feasible. In cases where such trials are not practical, investigators may turn to observational methods that can also produce reasonable conclusions. Observational studies, also called descriptive studies, chronicle existing environments and systems. The main goal of such studies is to describe in detail phenomena as they exist and evolve as the result of natural, as opposed to experimental, factors. Such studies allow the investigator to evaluate factors that may contribute to observable outcomes without introducing external changes for the sake of testing a hypothesis. In that sense, observational studies record “real-world” events as they unfold over time. Investigators may use observational study designs to detail workflow processes, to compare the impact of different environmental conditions among groups, and to record individuals' subjective impressions. Observational methods allow investigators to scrutinize single or multiple cases as examples of a given phenomenon; they can also provide insight into the environment in which phenomena of interest occur. Such methodologies include case reports, case series, cross sectional studies, workflow analyses, and qualitative evaluations such as surveys. For example, in the current issue of JAMIA, Johnson and Fitzhenry provide a series of in-depth workflow analyses outlining the processes healthcare providers follow in generating prescriptions.10 Likewise, Strayer and colleagues report on the time required for diffusion of new knowledge to various pharmacological reference texts and electronic resources.12 Both studies illustrate the rich detail that can be acquired using observational methods. For each report, a prospective controlled trial would not have been feasible. Investigators could not easily manipulate complex prescribing workflows, or influence the withdrawal of medication from the marketplace, simply to determine how information flow changed. While the lessons learned from these studies may not generalize well to other settings, and may be confounded by unmeasured factors, they nonetheless provide useful information. Healthcare informatics evaluation studies measure the influence of providing information on clinical outcomes. Among the many available methods and study designs, a small number are best suited to a given specific research task. Investigators should select the most robust, reproducible methodology for demonstrating associations between information-system-related interventions and clinical outcomes. They must further ensure that the chosen methodology is correctly applied to data collection and analysis. Informaticians must conduct carefully thought out and scientifically objective evaluations to prove that their systems neither cause harm (per “primum non nocere,” often included in the modern Hippocratic Oath) nor waste resources unnecessarily.

Previous investigators have defined clinical interface terminology as a systematic collection of health care-related phrases (terms) that supports clinicians' entry of patient-related information into computer programs, such as clinical "note capture" and decision support tools. Interface terminologies also can facilitate display of computer-stored patient information to clinician-users. Interface terminologies "interface" between clinicians' own unfettered, colloquial conceptualizations of patient descriptors and the more structured, coded internal data elements used by specific health care application programs. The intended uses of a terminology determine its conceptual underpinnings, structure, and content. As a result, the desiderata for interface terminologies differ from desiderata for health care-related terminologies used for storage (e.g., SNOMED-CT), information retrieval (e.g., MeSH), and classification (e.g., ICD9-CM). Necessary but not sufficient attributes for an interface terminology include adequate synonym coverage, presence of relevant assertional knowledge, and a balance between pre- and post-coordination. To place interface terminologies in context, this article reviews historical goals and challenges of clinical terminology development in general and then focuses on the unique features of interface terminologies.

Electronic health record (EHR) systems are increasingly being adopted in pediatric practices; however, requirements for integrated growth charts are poorly described and are not standardized in current systems. The authors integrated growth chart functionality into an EHR system being developed and installed in a multispecialty pediatric clinic in an academic medical center. During a three-year observation period, rates of electronically documented values for weight, stature, and head circumference increased from fewer than ten total per weekday, up to 488 weight values, 293 stature values, and 74 head circumference values (p<0.001 for each measure). By the end of the observation period, users accessed the growth charts an average 175 times per weekday, compared to 127 patient visits per weekday to the sites that most closely monitored pediatric growth. Because EHR systems and integrated growth charts can manipulate data, perform calculations, and adapt to user preferences and patient characteristics, users may expect greater functionality from electronic growth charts than from paper-based growth charts.

BACKGROUND: Unintended consequences of computerized patient care system interventions may increase resource use, foster clinical errors, and reduce users' confidence. OBJECTIVE: To evaluate three successive interventions designed to reduce serum magnesium test ordering through a care provider order entry system (CPOE). The second, modeled after a previously successful intervention, caused paradoxical increases in magnesium test ordering rates. DESIGN: A time-series analysis modeled weekly rates of magnesium test ordering, underlying trends, the impact of the three successive interventions, and the impact of potential covariates. The first intervention exhorted users to discontinue unnecessary tests recurring more than 72 hours into the future. The second displayed recent magnesium, calcium, and phosphorus test results, limited testing to one test instance per order, and provided education regarding appropriate indications for testing. The third targeted only magnesium ordering, displayed recent results, limited testing to one instance per order, summarized indications for testing, and required users to select an indication. PARTICIPANTS: Clinicians at Vanderbilt University Hospital, a 609-bed academic inpatient tertiary care facility, from 1998 through 2003. MEASUREMENTS: Weekly rates of new serum magnesium test orders, instances, and results. RESULTS: At baseline, there were 539 magnesium tests ordered per week. This decreased to 380 (p = 0.001) per week after the first intervention, increased to 491 per week (p < 0.001) after the second, and decreased to 276 per week (p < 0.001) after the third. CONCLUSION: A clinical decision support intervention intended to regulate testing increased test order rates as an unintended result of decision support. CPOE implementers must carefully design resource-related interventions and monitor their impact over time.

OBJECTIVE: Authors evaluated whether displaying context sensitive links to infrequently accessed educational materials and patient information via the user interface of an inpatient computerized care provider order entry (CPOE) system would affect access rates to the materials. DESIGN: The CPOE of Vanderbilt University Hospital (VUH) included "baseline" clinical decision support advice for safety and quality. Authors augmented this with seven new primarily educational decision support features. A prospective, randomized, controlled trial compared clinicians' utilization rates for the new materials via two interfaces. Control subjects could access study-related decision support from a menu in the standard CPOE interface. Intervention subjects received active notification when study-related decision support was available through context sensitive, visibly highlighted, selectable hyperlinks. MEASUREMENTS: Rates of opportunities to access and utilization of study-related decision support materials from April 1999 through March 2000 on seven VUH Internal Medicine wards. RESULTS: During 4,466 intervention subject-days, there were 240,504 (53.9/subject-day) opportunities for study-related decision support, while during 3,397 control subject-days, there were 178,235 (52.5/subject-day) opportunities for such decision support, respectively (p = 0.11). Individual intervention subjects accessed the decision support features at least once on 3.8% of subject-days logged on (278 responses); controls accessed it at least once on 0.6% of subject-days (18 responses), with a response rate ratio adjusted for decision support frequency of 9.17 (95% confidence interval 4.6-18, p < 0.0005). On average, intervention subjects accessed study-related decision support materials once every 16 days individually and once every 1.26 days in aggregate. CONCLUSION: Highlighting availability of context-sensitive educational materials and patient information through visible hyperlinks significantly increased utilization rates for study-related decision support when compared to "standard" VUH CPOE methods, although absolute response rates were low.

Care providers' adoption of computer-based health-related documentation ("note capture") tools has been limited, even though such tools have the potential to facilitate information gathering and to promote efficiency of clinical charting. The authors have developed and deployed a computerized note-capture tool that has been made available to end users through a care provider order entry (CPOE) system already in wide use at Vanderbilt. Overall note-capture tool usage between January 1, 1999, and December 31, 2001, increased substantially, both in the number of users and in their frequency of use. This case report is provided as an example of how an existing care provider order entry environment can facilitate clinical end-user adoption of a computer-assisted documentation tool-a concept that may seem counterintuitive to some.

OBJECTIVE: The Computerized Patient Record System is deployed at all 173 Veterans Affairs (VA) medical centers. Providers access clinical notes in the system from a note title menu. Following its implementation at the Nashville VA Medical Center, users expressed dissatisfaction with the time required find notes among hundreds of irregularly structured titles. The authors' objective was to develop a document-naming nomenclature (DNN) that creates informative, structured note titles that improve information access. DESIGN: One thousand ninety-four unique note titles from two VA medical centers were reviewed. A note-naming nomenclature and compositional syntax were derived. Compositional order was determined by user preference survey. MEASUREMENTS: The DNN was evaluated by modeling note titles from the Salt Lake City VA Medical Center (n=877), Vanderbilt University Medical Center (n=554), and the Mayo Clinic (n=42). A preliminary usability evaluation was conducted on a structured title display and sorting application. RESULTS: Classes of note title components were found by inspection. Components describe characteristics of the author, the health care event, and the organizational unit providing care. Terms were taken from VA medical center information systems and national standards. The DNN model accurately described 97 to 99 percent of note titles from the test sites. The DNN term coverage varied, depending on component and site. Users found the DNN title format useful and the DNN-based title sorting and note review application easy to learn and quick to use. CONCLUSION: The DNN accurately models note titles at five medical centers. Preliminary usability data indicate that DNN integration with title parsing and sorting software enhances information access.