OBJECTIVE: Developing targeted, culturally competent educational materials is critical for participant understanding of engagement in a large genomic study that uses computational pipelines to produce genome-informed risk assessments. MATERIALS AND METHODS: Guided by the Smerecnik framework that theorizes understanding of multifactorial genetic disease through 3 knowledge types, we developed English and Spanish infographics for individuals enrolled in the Electronic Medical Records and Genomics Network. Infographics were developed to explain concepts in lay language and visualizations. We conducted iterative sessions using a modified "think-aloud" process with 10 participants (6 English, 4 Spanish-speaking) to explore comprehension of and attitudes towards the infographics. RESULTS: We found that all but one participant had "awareness knowledge" of genetic disease risk factors upon viewing the infographics. Many participants had difficulty with "how-to" knowledge of applying genetic risk factors to specific monogenic and polygenic risks. Participant attitudes towards the iteratively-refined infographics indicated that design saturation was reached. DISCUSSION: There were several elements that contributed to the participants' comprehension (or misunderstanding) of the infographics. Visualization and iconography techniques best resonated with those who could draw on prior experiences or knowledge and were absent in those without. Limited graphicacy interfered with the understanding of absolute and relative risks when presented in graph format. Notably, narrative and storytelling theory that informed the creation of a vignette infographic was most accessible to all participants. CONCLUSION: Engagement with the intended audience who can identify strengths and points for improvement of the intervention is necessary to the development of effective infographics.

OBJECTIVE: This article reports on the alignment between the foundational domains and the delineation of practice (DoP) for health informatics, both developed by the American Medical Informatics Association (AMIA). Whereas the foundational domains guide graduate-level curriculum development and accreditation assessment, providing an educational pathway to the minimum competencies needed as a health informatician, the DoP defines the domains, tasks, knowledge, and skills that a professional needs to competently perform in the discipline of health informatics. The purpose of this article is to determine whether the foundational domains need modification to better reflect applied practice. MATERIALS AND METHODS: Using an iterative process and through individual and collective approaches, the foundational domains and the DoP statements were analyzed for alignment and eventual harmonization. Tables and Sankey plot diagrams were used to detail and illustrate the resulting alignment. RESULTS: We were able to map all the individual DoP knowledge statements and tasks to the AMIA foundational domains, but the statements within a single DoP domain did not all map to the same foundational domain. Even though the AMIA foundational domains and DoP domains are not in perfect alignment, the DoP provides good examples of specific health informatics competencies for most of the foundational domains. There are, however, limited DoP knowledge statements and tasks mapping to foundational domain 6-Social and Behavioral Aspects of Health. DISCUSSION: Both the foundational domains and the DoP were developed independently, several years apart, and for different purposes. The mapping analyses reveal similarities and differences between the practice experience and the curricular needs of health informaticians. CONCLUSIONS: The overall alignment of both domains may be explained by the fact that both describe the current and/or future health informatics professional. One can think of the foundational domains as representing the broad foci for educational programs for health informaticians and, hence, they are appropriately the focus of organizations that accredit these programs.

OBJECTIVE: Apply natural language processing (NLP) to Amazon consumer reviews to identify adverse events (AEs) associated with unapproved over the counter (OTC) homeopathic drugs and compare findings with reports to the US Food and Drug Administration Adverse Event Reporting System (FAERS). MATERIALS AND METHODS: Data were extracted from publicly available Amazon reviews and analyzed using JMP 16 Pro Text Explorer. Topic modeling identified themes. Sentiment analysis (SA) explored consumer perceptions. A machine learning model optimized prediction of AEs in reviews. Reports for the same time interval and product class were obtained from the FAERS public dashboard and analyzed. RESULTS: Homeopathic cough/cold products were the largest category common to both data sources (Amazon = 616, FAERS = 445) and were analyzed further. Oral symptoms and unpleasant taste were described in both datasets. Amazon reviews describing an AE had lower Amazon ratings (X2 = 224.28, P < .0001). The optimal model for predicting AEs was Neural Boosted 5-fold combining topic modeling and Amazon ratings as predictors (mean AUC = 0.927). DISCUSSION: Topic modeling and SA of Amazon reviews provided information about consumers' perceptions and opinions of homeopathic OTC cough and cold products. Amazon ratings appear to be a good indicator of the presence or absence of AEs, and identified events were similar to FAERS. CONCLUSION: Amazon reviews may complement traditional data sources to identify AEs associated with unapproved OTC homeopathic products. This study is the first to use NLP in this context and lays the groundwork for future larger scale efforts.

A discussion and debate on the American Medical Informatics Association's (AMIA) Ethical, Legal, and Social Issues (ELSI) Working Group listserv in 2021 raised important issues related to a forthcoming conference in Texas. Texas had recently enacted a restrictive abortion law and restricted voting rights. Several AMIA members advocated for a boycott of the state and the scheduled conference. The discussion led the AMIA Board of Directors to request that the organization's Ethics Committee provide general guidance for principle-based venue selection. This document recommends overarching principles for the venue selection for future AMIA events and conferences. Discussions by the AMIA Board, the Ethics Committee, and the ELSI Working Group informed these recommendations, and this document on guiding principles was approved by the AMIA Board of Directors in April 2022.

OBJECTIVE: This study aimed to understand the association between primary care physician (PCP) proficiency with the electronic health record (EHR) system and time spent interacting with the EHR. MATERIALS AND METHODS: We examined the use of EHR proficiency tools among PCPs at one large academic health system using EHR-derived measures of clinician EHR proficiency and efficiency. Our main predictors were the use of EHR proficiency tools and our outcomes focused on 4 measures assessing time spent in the EHR: (1) total time spent interacting with the EHR, (2) time spent outside scheduled clinical hours, (3) time spent documenting, and (4) time spent on inbox management. We conducted multivariable quantile regression models with fixed effects for physician-level factors and time in order to identify factors that were independently associated with time spent in the EHR. RESULTS: Across 441 primary care physicians, we found mixed associations between certain EHR proficiency behaviors and time spent in the EHR. Across EHR activities studied, QuickActions, SmartPhrases, and documentation length were positively associated with increased time spent in the EHR. Models also showed a greater amount of help from team members in note writing was associated with less time spent in the EHR and documenting. DISCUSSION: Examining the prevalence of EHR proficiency behaviors may suggest targeted areas for initial and ongoing EHR training. Although documentation behaviors are key areas for training, team-based models for documentation and inbox management require further study. CONCLUSIONS: A nuanced association exists between physician EHR proficiency and time spent in the EHR.

AMIA has a longstanding interest and a professional obligation to promote a strong ethical framework for its members and the field of biomedical and health informatics. This white paper presents the latest AMIA Code of Professional and Ethical Conduct. The original code was approved in 20071 by the AMIA Board of Directors and revised in 2013.2 Recognizing the need to regularly update the Code to ensure that it remains current and relevant, we present this document that constitutes a revision of and update to the third version, approved and published in the Journal of the American Medical Informatics Association in 2018.3 The code presented here remains an evolving document, with modifications expected as information technology, informatics, policy, and health care environments change over time. AMIA publishes on its web site the most recent version of the Code of Ethics as part of a process that seeks ongoing response from and involvement by AMIA members. Because the Code of Professional and Ethical Conduct (from now on “Code of Ethics”) is meant to be practical, applicable in real life, and easily understood, it is compact and uses general language. The AMIA Code of Ethics is not intended to be prescriptive or legislative; it is aspirational and extends beyond regulatory and legal obligations to provide the broad strokes of a set of important ethical principles pertinent to the field of biomedical and health informatics. The Code is organized around the common roles of AMIA members and the constituents they serve including patients, caregivers, colleagues and collaborators, clinicians, researchers, students, agencies, hospitals and practices, medical organizations, vendors, insurance companies, and others with whom they interact. The AMIA Board of Directors and the AMIA Ethics Committee encourage members to offer suggestions for improvements and changes. In this way, the Code will continue to evolve to best serve AMIA and the larger informatics community. AMIA members are professionally diverse,4,5 and include those who are, or are in training to be nurses, physicians, pharmacists, dentists, informaticians, computer scientists, analysts, implementation scientists, and other professionals. In many cases, these professions have their own codes of ethics.6–13 The International Medical Informatics Association, an international federation for which AMIA serves as the US membership organization, also has a revised “Code of Ethics for Health Information Professionals”.14 The AMIA Code of Ethics incorporates issues covered by other documents bearing on ethics and professional conduct: AMIA’s support for and efforts to incorporate and execute upon diversity, equity, inclusion, and accessibility goals and objectives throughout the organization.15 AMIA’s revised “Conflict of Interest Policy”, which governs the organization’s employees and leaders with regard to some of their financial and other interactions with outside entities.16 AMIA’s principles for selecting venues for conferences and other events, which affirm AMIA’s commitment to applying ethical principles and ensuring basic human rights when planning association events.17 AMIA’s “Meeting Anti-Harassment Policy”, which describes AMIA’s commitment to providing an atmosphere that is safe and welcoming to all members and supports learning and professional growth.18 AMIA’s principles for artificial intelligence (AI)19 and position on the appropriate development, use, and maintenance of adaptive clinical decision support.20 Members of the Ethics Committee are unanimous in their view that those who work in informatics, much as in other health professions, are duty-bound to embrace a patient-centered approach to their work, even if that work does not involve direct patient care or research involving human participants. As elsewhere in the health professions, vulnerable populations, historically and intentionally excluded/disinvested groups, and people with disabilities may reasonably expect additional considerations and support. The importance of professionalism and ethics has been recognized for millennia by health professionals and organizations,21 now including informaticians and information technology professionals. This code of ethics emphasizes AMIA’s commitment to adhere to and promote the highest standard of ethical and professional behavior. AMIA members acknowledge as their professional duty to uphold the following principles of and guidelines for ethical conduct. AMIA members are expected to know how to seek the advice of institutional ethics committees, AMIA’s Ethics Committee, or appropriate institutional review boards, as necessary. The following details address patient care, interactions with colleagues, responsibilities to employers, and roles regarding society and research. I. Key ethical guidelines regarding patients, guardians, and their authorized representatives (called here collectively “patients”) AMIA members involved in patient care should: Recognize that patients and their loved ones and caregivers have the right to know about the existence and use of electronic records containing their personal health and healthcare information, to access these records as written, and have the right to create and maintain their own personal health records and manage personal health information using a variety of platforms including mobile devices. In this context AMIA members should: Not mislead patients about the collection, use, or communication of their health information. Educate—when requested and within reason and the scope of their position—patients on the type, amount, and use of health information collected. Enable and—as appropriate, within reason and the scope of their position and in accord with independent ethical and legal standards—facilitate patients’ rights and ability to access, review, and correct their electronic health information, including clinicians’ notes. Recognize that patient-provided/generated health data, such as those collected on mobile devices and wearable devices, deserve the same diligence and protection as biomedical and health data gathered in the process of providing health care. Ensure that patients and their care team members are made aware of the role and use of AI and other complex automated tools that are not clearly apparent when such systems are involved in medical decision-making or care planning.19,20 Advocate and work as appropriate to ensure that protected health information (PHI),22 personally identifiable information (PII), and other biomedical data are acquired, recorded, stored, maintained, analyzed, transmitted, and communicated in an appropriately safe, reliable, secure, and confidential manner, and that such data management is consistent with applicable laws, local privacy and security policies, and accepted informatics standards. Never knowingly disclose PHI, PII, or biomedical or health data in violation of legal requirements or accepted local confidentiality practices, or in ways that are inconsistent with the explanation of data disclosure and use to the patient.23 AMIA members should understand that inappropriate disclosure of biomedical information can cause harm, and so should work to prevent such disclosures. AMIA members should avoid acquiring data through means that run the risk of, or fail to prevent, inappropriate disclosure. AMIA members should not accept, use, disseminate, or store data that they are aware were obtained in violation of applicable laws. Likewise, even if an action does not involve disclosure, one should not use or reuse—or through negligence permit the use of—patient information and data in ways inconsistent with the stated purposes, goals, or intentions of the patient or organization responsible for these data, except as appropriate for public health, previously approved and communicated research uses, quality improvement, or reporting as required under the law. Engage with patients, guardians, and their authorized representatives so as to support inclusion, promote equity, advance accessibility, and avoid bias and discrimination. II. Key ethical guidelines regarding colleagues AMIA members should: Endeavor, as appropriate, to support and foster colleagues’ and/or team members and their work, in a timely, respectful, and conscientious way to support their roles in healthcare and/or research and education. Support and foster the efforts of patients to be actively involved in the collection, management, and curation of their health data. Advise colleagues and others, as appropriate, about actual or potential information or systems issues (including system flaws, defects, usability or performance issues, etc.) that negatively affect patient safety, privacy, data security, or health outcomes or could hinder colleagues’ abilities to delegate responsibilities to patients, other colleagues, involved institutions, or other stakeholders. Actively support the inclusion of all professional colleagues and promote a diverse and inclusive environment in which all individuals have equitable access to resources, educational opportunities, and opportunities for professional advancement.15 An AMIA member in any leadership position should: Be familiar with these guidelines and their applicability to their practice, unit, or organization. Communicate as appropriate about these ethical guidelines to those they lead. Strive to promote familiarity with, and use of, these ethical guidelines. Use AMIA position statements to guide organizational decision-making with regard to diversity, equity, inclusion, and accessibility initiatives,15 including selection of event locations.17 Promote transparent and equitable decision-making among AMIA professional staff, volunteer member leaders, and others with whom they engage. Never allow personal political views or ideological stances to interfere with or impede their ability to represent AMIA and advocate for it. AMIA leaders who may pose a reputational liability to the organization due to criminal convictions should declare this information and be prepared to stand down from leadership positions. Members who may pose a reputational liability should recuse themselves from leadership positions. III. Key ethical guidelines regarding institutions, employers, business partners, and clients (called here collectively “employers”) AMIA members should: Understand their duties and obligations to current and former employers and fulfill them to the best of their abilities within the bounds of ethical and legal norms. Understand and appreciate that employers have legal and ethical rights and obligations, including those related to intellectual property. Understand and respect the obligations of their employers and comply with local policies and procedures to the extent that they do not violate ethical and legal norms. Consider the tradeoffs that occur with the configuration and use of technologies (eg, decision support systems) before implementation, and monitor, measure, and manage results when the optimal approach is unclear. Inform the employer and act in accordance with ethical-legal mandates and patient rights when employer actions, policies, or procedures would violate actual or understood ethical or legal obligations, contracts, or other agreements made with patients. Maintain a safe and high-quality environment even while implementing innovation, recognizing that all changes in a complex adaptive environment generate unanticipated consequences and potential harm. IV. Key ethical guidelines regarding society and regarding research AMIA members involved in research should: Be aware of the Declaration of Helsinki (Ethical Principles for Medical Research Involving Human Subjects), the Nuremberg Report, and the Belmont Report, which should guide all human subjects research, including research that involves users of informatics tools and interventions as participants (eg, workflow analysis studies, evaluation of clinical decision support systems, patient care innovations, analysis, etc.).24–27 Recognize that duty and care to individuals such as patients and colleagues exist regardless of whether such responsibilities are acknowledged by institutional review boards, vendors, and others involved in informatics activities. Be mindful and respectful of the social or public health implications of their work, ensuring that the greatest good for society is balanced by ethical obligations to individual patients. Uphold standards for publication and authorship, including the International Committee of Medical Journal Editors’ “Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals.”28 These recommendations are paralleled by the editorial policies for the past29 and current30 publishers of the Journal of the American Medical Informatics Association, as well as the publisher of Applied Clinical Informatics31 and their open access companion journals. Such efforts include, but are not limited to, avoiding any plagiarism or self-plagiarism or other misrepresentations of the truth in the publication of research and other work. Disseminate new knowledge—both positive and negative findings—expeditiously, to allow the field to advance and to permit others to take advantage of novel discoveries and understanding to improve patient care. Strive, as appropriate, in the context of one’s position to foster the generation of knowledge and biomedical advances through appropriate support for ethical and institutionally approved research efforts facilitated through informed consent and robust data governance, including disclosure processes and procedures, particularly when third-party entities not meeting the definition of business associates are involved. Know and abide by the applicable governmental regulations and institutional policies that define ethical research in their professional environment. V. General professional and ethical guidelines AMIA members should: Maintain competence as informatics professionals: Obtain applicable continuing education and be dedicated to a culture of lifelong learning and self-improvement. Recognize technical and ethical limitations and seek consultation when needed, particularly in ethically conflicting situations. Contribute to the education and mentoring of students, early-career members, and others, as appropriate. Promote a culture of diversity, equity, inclusion, and accessibility in their work and professional conduct. Strive to encourage the adoption of informatics approaches supported by adequate evidence to improve health and healthcare; and to encourage and support efforts to improve the amount and quality of such evidence. Treat all individuals with respect and not discriminate against anyone based on age, race, ethnicity, gender identity, disability (visible or invisible), national origin, sexual orientation, religion, or residency status. Be mindful that their work and actions reflect on the profession and on AMIA. The Code’s authors are aware that all professionals will, from time to time, find themselves in situations shaped by what has been called “dual agency” or “multiple agency”. In these circumstances, a professional encounters conflicting commitments, duties, or loyalties. An informatics professional may have conflicting duties to patients, to colleagues, to society, and to an employer. No code of ethics can resolve contradictions, but a well-crafted code may enable priorities to be set down explicitly and so provide a guide to action. In addition to this Code of Ethics, the AMIA’s Ethics Committee and its Conflict of Interest Panel are primary resources for members who find themselves in ethically unclear or challenging situations. For scholarship and education related to ethical issues in the broader field of medical information, the AMIA Ethical, Legal, and Social Issues (ELSI) Working Group serves as a community forum for members. As a matter of personal and professional integrity, adherence to the principles laid out here is expected of all who have the privilege of serving in the field of biomedical and health informatics. Those whose skills allow them to contribute in one way or another to the health of individuals and populations carry important responsibilities. This code of ethics provides guidance about how informaticians may best do so. All authors participated in the revision, review, and approval of this manuscript. Because this work is a revision of AMIA’s Code of Professional and Ethical Conduct 2018, no author can be considered to be responsible for the conception or design of the work. The authors and the AMIA Ethics Committee would like to thank the AMIA Board of Directors for its continuing interest in refining and publishing these guidelines. Phyllis Burchman, AMIA’s former Director of Office Operations and Human Resources, long provided invaluable support to the Ethics Committee in its work. Members of the AMIA Ethics Committee who contributed to the third version of the code in 2018 and are not otherwise listed here include Peter Embi, Harold Lehmann, Sarah A. Maulden, Kyle A. McGregor, and Enrique Terrazas. This version of the code also owes much to the members of AMIA’s Ethical, Legal, and Social Issues (ELSI) Working Group. None declared.

AMIA has a longstanding interest in and a professional obligation to promote a strong ethical framework for the field of biomedical and health informatics. This white paper presents the latest AMIA Code of Professional and Ethical Conduct. The original Code was approved in 20071 by the AMIA Board of Directors. Recognizing the need to update the Code to ensure that it remains current and relevant, this document constitutes a revision of and update to the second code, approved in 2012 and published in the Journal of the American Medical Informatics Association in 2013.2 The code presented here remains an evolving document, with modifications expected as the information technology, informatics, and healthcare environments change over time. AMIA will publish on its web site the most recent version of the Code of Ethics as part of a process that seeks ongoing response from and involvement by AMIA members. Because the Code is meant to be practical, applicable in real life, and easily understood, it is compact and uses general language. The AMIA Code of Ethics is not intended to be prescriptive or legislative; it is aspirational and extends beyond regulatory and legal obligations to provide the broad strokes of a set of important ethical principles pertinent to the field of biomedical and health informatics. The Code is organized around the common roles of AMIA members and the constituents they serve, including patients, caregivers, clinicians, researchers, students, agencies, hospitals and practices, medical organizations, vendors, insurance companies, and others with whom they interact. The AMIA Board and the AMIA Ethics Committee encourage members to offer suggestions for improvements and changes. In this way, the Code will continue to evolve to best serve AMIA and the larger informatics community. The Code’s authors are aware that all professionals will, from time to time, find themselves in situations shaped by what has been called “dual agency” or “multiple agency.” In these circumstances, a professional encounters conflicting commitments, duties, or loyalties. An informatics professional may have conflicting duties to patients, to colleagues, to society, and to an employer. Few, if any, codes of ethics are nimble enough to provide guidance in such situations. Further, AMIA’s Ethics Committee is a resource to members who find themselves in ethically unclear or challenging situations. AMIA members may contact the AMIA Ethics Committee, which can provide guidance in some circumstances. AMIA members are professionally diverse,3,4 and include those who are, or are in training to be, nurses, physicians, pharmacists, dentists, informaticians, computer scientists, and others. In many cases, these professions have their own codes of ethics.5–12 The International Medical Informatics Association, an international federation for which AMIA serves as the U.S. membership organization, also has a revised “Code of Ethics for Health Information Professionals.”13 This document incorporates issues covered by other documents and laws bearing on ethics and professional conduct: AMIA’s “Conflict of Interest Policy,” which governs the organization’s employees and leaders in regard to some of their financial and other interactions with outside entities.14 AMIA’s “Meeting Anti-Harassment Policy,” which describes AMIA’s commitment to providing an atmosphere that is welcoming to all members and supports learning and professional growth.15 The International Committee of Medical Journal Editors’ “Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals.”16 This document is widely accepted as identifying standards for publication and authorship, and is paralleled by the editorial policies for the past17 and current18 publishers of the Journal of the American Medical Informatics Association, as well as the publisher of Applied Clinical Informatics.19 Privacy laws. Several sections herein address patient privacy or the rights of patients to view and control access to their health information. These sections are intended to parallel and make explicit duties under the law. In the United States, for instance, the Privacy Rule under the Health Insurance Portability and Accountability Act20 lays out many duties for those who are entrusted with health information. Many other countries have similar laws to protect patient data. Informatics professionals are expected to be familiar with and follow the laws governing their practice. Members of the Ethics Committee are unanimous in their view that those who work in informatics, much as in other health professions, are duty-bound to embrace a patient-centered approach to their work, even if that work does not involve direct patient care or human subjects research. As elsewhere in the health professions, vulnerable populations or those with special needs may be entitled to additional considerations. The importance of professionalism and ethics has been recognized for millennia by health professionals and organizations,21 now including information technology professionals. This code of ethics emphasizes AMIA’s commitment to adhere to and promote the highest standard of ethical and professional behavior. AMIA members acknowledge as their professional duty to uphold the following principles of and guidelines for ethical conduct. AMIA members are expected to know to seek the advice of institutional ethics committees, AMIA’s Ethics Committee, or appropriate institutional review boards, as necessary. The following code details address patient care, interactions with colleagues, responsibilities to employers, and roles regarding society and research. I. Key ethical guidelines regarding patients, guardians, and their authorized representatives (called here collectively “patients”) AMIA members involved in patient care should: A. Recognize that patients and their loved ones and caregivers have the right to know about the existence and use of electronic records containing their personal healthcare information, and have the right to create and maintain their own personal health records and manage personal health information using a variety of platforms including mobile devices. In this context AMIA members should: Not mislead patients about the collection, use, or communication of their healthcare information. Enable and — as appropriate, within reason and the scope of their position and in accord with independent ethical and legal standards — facilitate patients’ rights and ability to access, review, and correct their electronic health information. Recognize that patient-provided/generated health data, such as those collected on mobile devices, deserve the same diligence and protection as biomedical and health data gathered in the process of providing health care. B. Advocate and work as appropriate to ensure that protected health information (PHI),20 personally identifiable information (PII), and other biomedical data are transmitted, acquired, recorded, stored, maintained, analyzed, and communicated in an appropriately safe, reliable, secure, and confidential manner, and that such data management is consistent with applicable laws, local privacy and security policies, and accepted informatics standards. C. Never knowingly disclose PHI, PII, or biomedical or health data in violation of legal requirements or accepted local confidentiality practices, or in ways that are inconsistent with the explanation of data disclosure and use to the patient. AMIA members should understand that inappropriate disclosure of biomedical information can cause harm, and so should work to prevent such disclosures. AMIA members should avoid acquiring data through means that run the risk of, or fail to prevent, inappropriate disclosure. Likewise, even if an action does not involve disclosure, one should not use — or through negligence permit the use of — patient information and data in ways inconsistent with the stated purposes, goals, or intentions of the patient or organization responsible for these data, except as appropriate for public health, previously approved and communicated research uses, or reporting as required under the law. II. Key ethical guidelines regarding colleagues AMIA members should: A. Endeavor, as appropriate, to support and foster colleagues’ and/or team members’ work, in a timely, respectful, and conscientious way to support their roles in healthcare and/or research and education. B. Support and foster the efforts of patients to be actively involved in the collection, management, and curation of their health data. C. Advise colleagues and others, as appropriate, about actual or potential information or systems issues (including system flaws, bugs, usability issues, etc.) that negatively affect patient safety, privacy, data security, or outcomes or could hinder colleagues’ ability to delegate responsibilities to patients, other colleagues, involved institutions, or other stakeholders. D. If a leader, an AMIA member should: Be familiar with these guidelines and their applicability to their practice, unit, or organization. Communicate as appropriate about these ethical guidelines to those they lead. Strive to promote familiarity with, and use of, these ethical guidelines. III. Key ethical guidelines regarding institutions, employers, business partners, and clients (called here collectively “employers”) AMIA members should: A. Understand their duties and obligations to current and former employers and fulfill them to the best of their abilities within the bounds of ethical and legal norms. B. Understand and appreciate that employers have legal and ethical rights and obligations, including those related to intellectual property. Understand and respect the obligations of their employers, and comply with local policies and procedures to the extent that they do not violate ethical and legal norms. Consider the tradeoffs that occur with the configuration and use of technologies (eg, decision support systems) before implementation, and monitor and manage results when the optimal approach is unclear. C. Inform the employer and act in accordance with ethico-legal mandates and patient rights when employer actions, policies, or procedures would violate ethical or legal obligations, contracts, or other agreements made with patients. Maintain a safe and high-quality environment even while implementing innovation, recognizing that all changes in a complex adaptive environment generate unanticipated consequences and potential harm. IV. Key ethical guidelines regarding society and regarding research AMIA members involved in research should: A. Be aware of the Declaration of Helsinki (Ethical Principles for Medical Research Involving Human Subjects), which should guide all human subject research, including research that involves users of informatics tools and interventions as human subjects (eg, workflow analysis studies, clinical decision support systems analysis, patient care innovations, analysis, etc.).22,23 Recognize that duty and care to colleagues exist regardless of whether such responsibilities are acknowledged by institutional review boards, vendors, and others involved in informatics activities. B. Be mindful and respectful of the social or public health implications of their work, ensuring that the greatest good for society is balanced by ethical obligations to individual patients. C. Avoid any plagiarism or self-plagiarism or other misrepresentations of the truth in the publication of research and other work. D. Disseminate new knowledge — both positive and negative — expeditiously, to allow the field to advance and to permit others to take advantage of novel discoveries to improve patient care. E. Strive as appropriate in the context of one’s position to foster the generation of knowledge and biomedical advances through appropriate support for ethical and institutionally approved research efforts facilitated through informed consent and disclosure processes and procedures, particularly when third-party entities not meeting the definition of business associates are involved. F. Know and abide by the applicable governmental regulations and local policies that define ethical research in their professional environment. V. General professional and ethical guidelines AMIA members should: A. Maintain competence as informatics professionals: Obtain applicable continuing education and be dedicated to a culture of lifelong learning and improvement; Recognize technical and ethical limitations and seek consultation when needed, particularly in ethically conflicting situations; Contribute to the education and mentoring of students, junior members, and others, as appropriate; Promote a culture of inclusivity in their work and professional conduct. B. Strive to encourage the adoption of informatics approaches supported by adequate evidence to improve health and healthcare; and to encourage and support efforts to improve the amount and quality of such evidence. C. Be mindful that their work and actions reflect on the profession and on AMIA. As a matter of personal and professional integrity, adherence to the principles laid out here is expected of all who have the privilege of serving in the field of biomedical and health informatics. Those whose skills allow them to contribute in one way or another to the health of individuals and populations carry important responsibilities, and this Code of Ethics delineates how informaticians may best do so. None. Not commissioned; not peer reviewed. Conflict of interest statement. None. The authors and the AMIA Ethics Committee would like to thank the AMIA Board of Directors for its continuing interest in refining and publishing these guidelines. Phyllis Burchman, AMIA’s Director of Office Operations and Human Resources, provided invaluable support to the Ethics Committee in its work. Members of the AMIA Ethics Committee who contributed to the second version of the code in 2012 and are not otherwise listed here include Samantha Adams, Robert Hsiung, John Hurdle, and Dixie A. Jones. This version of the code also owes much to the members of AMIA’s Ethical, Legal, and Social Issues (ELSI) Working Group.

This White Paper presents the foundational domains with examples of key aspects of competencies (knowledge, skills, and attitudes) that are intended for curriculum development and accreditation quality assessment for graduate (master's level) education in applied health informatics. Through a deliberative process, the AMIA Accreditation Committee refined the work of a task force of the Health Informatics Accreditation Council, establishing 10 foundational domains with accompanying example statements of knowledge, skills, and attitudes that are components of competencies by which graduates from applied health informatics programs can be assessed for competence at the time of graduation. The AMIA Accreditation Committee developed the domains for application across all the subdisciplines represented by AMIA, ranging from translational bioinformatics to clinical and public health informatics, spanning the spectrum from molecular to population levels of health and biomedicine. This document will be periodically updated, as part of the responsibility of the AMIA Accreditation Committee, through continued study, education, and surveys of market trends.

OBJECTIVE: Informatics programs need assurance that their curricula prepare students for intended roles as well as ensuring that students have mastered the appropriate competencies. The objective of this study is to describe a method for using assessment data to identify areas for curriculum, student selection, and assessment improvement. MATERIALS AND METHODS: A multiple-choice examination covering the content in the Commission for Health Accreditation of Informatics and Information Management Education curricular facets/elements was developed and administered to 2 cohorts of entering students prior to the beginning of the program and to the first cohort after completion of the first year's courses. The reliability of the examination was assessed using Cronbach's alpha. Content validity was assessed by having 2 raters assess the match of the items to the Commission for Health Accreditation of Informatics and Information Management Education requirements. Construct validation included comparison of exam performance of instructed vs uninstructed students. Criterion-related validity was assessed by examining the relationship of background characteristics to exam performance and by comparing examination performance to graduate Grade Point Average (GPA). RESULTS: Reliability of the examination was 0.91 and 0.82 (Cohort 1 pre/post-tests) and 0.43 (Cohort 2 pretest). Both raters judged 76% of the test items as appropriate. There were statistically significant differences between the instructed (Cohort 1 post-test) and uninstructed (Cohort 2 pretest) students (t = 2.95 P < .01), as well as between the Cohort 1 pre/post-tests (t = 6.52, P < .001). Neither the background variables nor the graduate GPA were significantly correlated with the examination scores. CONCLUSION: We found that the examination had generally good psychometric properties and the exceptions could be used to identify areas for curriculum and assessment improvement.

OBJECTIVE: As part of the Heath Information Technology for Economic and Clinical Health (HITECH) Act, the Office of the National Coordinator for Health Information Technology (ONC) implemented its Workforce Development Program, which included initiatives to train health information technology (HIT) professionals in 12 workforce roles, half of them in community colleges. To achieve this, the ONC tasked five universities with established informatics programs with creating curricular materials that could be used by community colleges. The five universities created 20 components that were made available for downloading from the National Training and Dissemination Center (NTDC) website. This paper describes an evaluation of the curricular materials by its intended audience of educators. METHODS: We measured the quantity of downloads from the NTDC site and administered a survey about the curricular materials to its registered users to determine use patterns and user characteristics. The survey was evaluated using mixed methods. Registered users downloaded nearly half a million units or components from the NTDC website. We surveyed these 9835 registered users. RESULTS: 1269 individuals completed all or part of the survey, of whom 339 identified themselves as educators (26.7% of all respondents). This paper addresses the survey responses of educators. DISCUSSION: Successful aspects of the curriculum included its breadth, convenience, hands-on and course planning capabilities. Several areas were identified for potential improvement. CONCLUSIONS: The ONC HIT curriculum met its goals for community college programs and will likely continue to be a valuable resource for the larger informatics community in the future.

AMIA, as other professional societies, has a long-standing interest in promoting a strong ethical framework for its membership. This white paper presents the latest AMIA Code of Professional and Ethical Conduct. It was approved in November of 2011 by the AMIA Board of Directors. This document constitutes a revision of, and update to, the first code, approved and published in J Am Med Inform Assoc1 in 2007. In an effort to keep pace with the field's vitality, the code presented here is intended to be a dynamic document, and will continue to evolve as AMIA and the field itself evolve. AMIA will publish on its web site this version of the code as part of a process that seeks ongoing response from, and involvement by, AMIA members. The code is meant to be practical and easily understood, so it is compact and uses general language. Unlike the ethics codes of some professional societies, the AMIA code is not intended to be prescriptive or legislative; it is aspirational, and as such, provides the broad strokes of a set of important ethical principles especially pertinent to the field of biomedical and health informatics. The code is organized around the common roles of AMIA members and the constituents they serve—including patients, students, and others—and with whom they interact. The AMIA Board and the AMIA Ethics Committee encourage members to offer suggestions for improvements and other changes. In this way, the code will continue to progress and best serve AMIA and the larger informatics community.

At the 2011 American College of Medical Informatics (ACMI) Winter Symposium we studied the overlap between health IT and economics and what leading healthcare delivery organizations are achieving today using IT that might offer paths for the nation to follow for using health IT in healthcare reform. We recognized that health IT by itself can improve health value, but its main contribution to health value may be that it can make possible new care delivery models to achieve much larger value. Health IT is a critically important enabler to fundamental healthcare system changes that may be a way out of our current, severe problem of rising costs and national deficit. We review the current state of healthcare costs, federal health IT stimulus programs, and experiences of several leading organizations, and offer a model for how health IT fits into our health economic future.

The current commercial health information technology (HIT) arena encompasses a number of competing firms that provide electronic health applications to hospitals, clinical practices, and other healthcare-related entities. Such applications collect, store, and analyze patient information. Some vendors incorporate contract language whereby purchasers of HIT systems, such as hospitals and clinics, must indemnify vendors for malpractice or personal injury claims, even if those events are not caused or fostered by the purchasers. Some vendors require contract clauses that force HIT system purchasers to adopt vendor-defined policies that prevent the disclosure of errors, bugs, design flaws, and other HIT-software-related hazards. To address this issue, the AMIA Board of Directors appointed a Task Force to provide an analysis and insights. Task Force findings and recommendations include: patient safety should trump all other values; corporate concerns about liability and intellectual property ownership may be valid but should not over-ride all other considerations; transparency and a commitment to patient safety should govern vendor contracts; institutions are duty-bound to provide ethics education to purchasers and users, and should commit publicly to standards of corporate conduct; and vendors, system purchasers, and users should encourage and assist in each others' efforts to adopt best practices. Finally, the HIT community should re-examine whether and how regulation of electronic health applications could foster improved care, public health, and patient safety.

The American College of Medical Informatics (ACMI), is “a college of elected fellows from the United States and abroad who have made significant and sustained contributions to the field of medical informatics” and is a component of the American Medical Informatics Association (AMIA). Over the last few years in its winter symposium, ACMI has discussed a variety of issues related to the adoption and deployment of electronic health records (EHRs). Beginning in winter 2004, ACMI addressed what would be needed to promote greater adoption of electronic health records. The recommendations led to AMIA's “Got EHR?” initiative (www.amia.org/gotehr) and a series of articles summarizing the ACMI participants' recommendations.1–4 In 2005, the ACMI symposium focused on personal health records (PHRs), and the resulting article outlined recommendations for the deployment of PHRs, an area that is now showing increasing development nationwide.5 In 2006, with the growing interest in clinical decision support (see AMIA's Clinical Decision Support Roadmap initiative http://www.amia.org/inside/initiatives/cds/), ACMI addressed the knowledge management that would be needed for effective broad-based use of clinical decision support systems. These recommendations were presented at the 2006 AHRQ Patient Safety and Healthcare IT Conference (http://healthit.ahrq.gov/portal/server.pt/gateway/PTARGS_0_4450_132264_0_0_18/Berner%204-IV%20Aud.ppt). Over the same period of time nationally, the momentum for the adoption of EHRs has accelerated. A National Coordinator for Health Information Technology was appointed; the Agency for Healthcare Research and Quality (AHRQ) developed a series of funding initiatives related to research and implementation of information technology in health care; and the American Health Information Community (AHIC) was formed. AHRQ, in partnership with the Office of the National Coordinator, also funded pilot studies on the development of a National Health Information Network and Regional Health Information Organizations (RHIOs) formed in many parts of the country. Suddenly, the deployment of EHRs was clearly on the national radar and adoption was accelerating. The next logical topics for ACMI to address, now that adoption and implementation were increasing, were the issues involved in implementation of these EHR systems. The 2007 ACMI winter symposium focused on recommendations related to implementation. However, along with the growing interest nationally in the implementation of clinical systems for patient care, there was recognition of a need to implement information systems to promote clinical and translational research. The National Institutes of Health Clinical and Translation Science Award (CTSA) program (www.ctsaweb.org) mandated that each research center have a strong biomedical informatics component whose role would include integration of clinical, research, and bioinformatics systems. Systems for translational scientific research have their own implementation challenges, and the 2007 symposium also provided the opportunity for ACMI members, many of whom are in leadership positions within their institution's CTSA centers, to share their experiences. As a result, AMIA has added a focus on clinical research informatics to its areas of interest and is sponsoring a new translational bioinformatics conference; it is likely that in the future the area of translational and clinical research informatics will grow in importance. Thus, the 2007 ACMI winter symposium focused on the challenges of implementing both electronic health records and clinical and translational research systems. The set of articles in this special section of the Journal grew out of the 2007 ACMI meeting. Each article evolved from a presentation at the meeting by one of the authors, which was followed by participant discussion and recommendations. Each manuscript itself was finalized by its authors, several of whom had also contributed to the original presentation. While the papers represent the viewpoints of their individual authors and are not ACMI or AMIA position papers, the input from the ACMI participants helped shape their final form. In keeping with the trend nationally for team science for biomedical research, the article by Lorenzi et al. proposes that there should be a similar team science approach to research to develop best practices for implementation.6 Although the focus of Lorenzi et al. is on clinical systems, their approach could be applied to the development and implementation of the research systems as well. In fact, the CTSA awardees have formed a National Informatics Steering Committee to share best practices and address common problems. The second article, by McGowan et al., discusses best practices for formative evaluation of the implementation of EHR systems.7 Again, the principles and approaches they advocate can be adapted for the needed formative evaluation of the implementation of any new information systems, including those that will be developed in the context of the CTSA awards. The last article, by Ash et al., directly addresses the people and organizational issues in the implementation of both clinical and research systems.8 Recognizing and managing people and organizational challenges are crucial for successful implementations. Although many of these issues have been recognized and studied in clinical system implementations, they are equally important to implementations of new research systems. As Chair of ACMI's Scientific Affairs Committee from 2005–2007, I want to express my appreciation to Jeffrey Williamson, who provided support for the committee and its meetings. It was my privilege to work with an excellent program planning committee: Patricia Abbott, Michael Ackerman, Joan Ash, R. Scott Evans, Mark Frisse, Julie McGowan, Kevin Johnson, and Blackford Middleton, and with ACMI presidents Paul Clayton, Judy Ozbolt, and Dan Masys. In addition to the specific presentations that formed the background for the articles in this special section, presentations at the 2007 meeting by Charles Friedman, R. Scott Evans, Michael Kahn, William Hersh, Christopher Chute, Jack Smith, Michael Becich, Dan Masys, and Don Detmer helped shaped the dialogue that led to their creation. The many participants in these meetings over the years cannot be acknowledged by name, but their insights and willingness to share their expertise contributed significantly to these articles and the others that preceded them.

OBJECTIVE: The authors developed and evaluated a rating scale, the Attitudes toward Handheld Decision Support Software Scale (H-DSS), to assess physician attitudes about handheld decision support systems. DESIGN: The authors conducted a prospective assessment of psychometric characteristics of the H-DSS including reliability, validity, and responsiveness. Participants were 82 Internal Medicine residents. A higher score on each of the 14 five-point Likert scale items reflected a more positive attitude about handheld DSS. The H-DSS score is the mean across the fourteen items. Attitudes toward the use of the handheld DSS were assessed prior to and six months after receiving the handheld device. STATISTICS: Cronbach's Alpha was used to assess internal consistency reliability. Pearson correlations were used to estimate and detect significant associations between scale scores and other measures (validity). Paired sample t-tests were used to test for changes in the mean attitude scale score (responsiveness) and for differences between groups. RESULTS: Internal consistency reliability for the scale was alpha = 0.73. In testing validity, moderate correlations were noted between the attitude scale scores and self-reported Personal Digital Assistant (PDA) usage in the hospital (correlation coefficient = 0.55) and clinic (0.48), p < 0.05 for both. The scale was responsive, in that it detected the expected increase in scores between the two administrations (3.99 (s.d. = 0.35) vs. 4.08, (s.d. = 0.34), p < 0.005). CONCLUSION: The authors' evaluation showed that the H-DSS scale was reliable, valid, and responsive. The scale can be used to guide future handheld DSS development and implementation.

For over thirty years, there have been predictions that the widespread clinical use of computers was imminent. Yet the "wave" has never broken. In this article, two broad time periods are examined: the 1960's to the 1980's and the 1980's to the present. Technology immaturity, health administrator focus on financial systems, application "unfriendliness," and physician resistance were all barriers to acceptance during the early time period. Although these factors persist, changes in clinicians' economics, more computer literacy in the general population, and, most importantly, changes in government policies and increased support for clinical computing suggest that the wave may break in the next decade.

As we move toward an era when health information is more readily accessible and transferable, there are several issues that will arise. This article addresses the challenges of information filtering, context-sensitive decision support, legal and ethical guidelines regarding obligations to obtain and use the information, aligning patient and health professionals' expectations in regard to the use and usefulness of the information, and enhancing data reliability. The authors discuss the issues and offer suggestions for addressing them.

In 1996 in an editorial on evaluation of decision support systems, Miller proposed that the bottom line in evaluating clinical decision support systems (CDSSs) should be “whether the user plus the system is better than the unaided user with respect to a specified task….”1 Since 1996, several studies have examined that issue, and, yet, there is still disagreement on the way to operationalize Miller's proposition. In this issue of the Journal, Ramnarayan et al.2 describe a variety of metrics to evaluate the performance of a new pediatric diagnostic program, ISABEL. In a previous issue, Fraser et al.3 also described metrics to evaluate a heart disease program, the HDP. Both Ramnarayan et al. and Fraser et al. discussed how their measures compared with the earlier measures used by Berner et al.4 and Friedman et al.5 to evaluate other diagnostic programs. Why should it be so difficult to agree on a reasonable metric for evaluating these systems? Those of us who have struggled with this issue in our research have come to appreciate some of the difficulties that may not be immediately obvious in the published literature, but are important to articulate. Many of these issues are not unique to the diagnostic programs, but are a challenge in evaluating any CDSS. However, diagnostic programs are particularly challenging because, as Ramnarayan et al. indicate, diagnostic programs should influence both the diagnosis and the management plans. With that in mind, and with Miller's injunction to focus on evaluating how the system and clinician work together, I would like to discuss the problems that arise with the different “gold standards” that researchers have used and also would like to offer suggestions for researchers and developers of diagnostic CDSS. Most researchers have included in their metrics the production of the “correct” diagnosis by either the CDSS or the clinicians after using the CDSS. Many have looked at the rank of the correct diagnosis on the differential, assuming that more highly ranked is better. On an intuitive basis, the CDSS's “getting the right answer” should be a good standard to use to judge the quality of a CDSS, and failure to do so has led some to dismiss the worth of these systems.6 However, there are problems with this criterion. It could as well be argued that this is not a good criterion, since a definitive correct diagnosis is not always needed to initiate workup or treatment. Also, as Ramnarayan et al. point out, in real life the information needed to be certain of the correct diagnosis is unlikely to be known at the time that decision support is sought. In addition, if the correct diagnosis is a very rare one, it is likely that other diagnoses will be in a more prominent position on the list of both the CDSS and the clinician. This leads to a paradox; the highly ranked diagnoses are likely to already be considered by the clinician, while the lower ranked ones may not seem credible. In recognition of some of the problems of relying entirely on the use of the correct diagnosis as a gold standard, Ramnarayan et al. and other researchers also have included a measure of the quality of the output of the CDSS, and/or the clinicians' differential, and have relied on expert opinion to determine the “goodness” of the differential. There are several problems with this approach. If the experts use the full case data with definitive test results to judge the quality of a differential when the user and/or the CDSS did not have all of that data, there is a risk of both hindsight bias and underestimation of the quality of the performance of the CDSS. Fraser et al. discussed this possibility in their study and also noted another problem, that there are often disagreements among experts. To avoid these problems, Ramnarayan et al. had the experts develop their own differential diagnoses and judge the appropriateness of the diagnoses with only the initial data and without knowledge of the “correct” diagnosis. However, although Ramnarayan et al. note that the final case diagnosis was always included in the list of appropriate diagnoses and was almost always ranked highly, the reliance on the collective opinion of experts is not a substitute for definitive data. If diagnosis is an intermediate step toward appropriate patient management, maybe a focus on how a diagnostic decision support system influences the clinician's management is a better focus than simply focusing on the correctness or quality of the CDSS diagnostic suggestions or the clinician's diagnoses. Ramnarayan et al. have included a measure of management, as well as diagnostic, quality and found a moderately positive relationship between the two measures. While examining the impact of a diagnostic system on patient management is important to do, the same clinical scenarios (whether simulated, real, consecutive cases, or particular diagnostic challenges) may not be appropriate to adequately test both kinds of suggestions. For instance, my colleagues and I examined the impact of a clinician's considering the correct diagnosis on ordering the definitive diagnostic procedure. We found that for some cases, if the correct diagnosis were not considered, the correct procedure would not be done. For others, a diagnosis of “something weird neurologic” was sufficient to lead to ordering the computed tomography (CT) scan or magnetic resonance imaging (MRI), which would ultimately provide the diagnosis.7 The neurologic case was very difficult diagnostically, but was not a sensitive measure of management appropriateness. Some researchers consider users' own judgment of helpfulness of the CDSS as the appropriate metric to use to judge its worth. This measure, too, is fraught with difficulty. Less clinically sophisticated users may be the ones most in need of decision support and, in fact, may perceive the CDSS to be quite helpful, but they also may be the least able to accurately judge their own knowledge and the appropriateness of the CDSS suggestions. It has been suggested that users vote with their feet (or at least with their fingers) as to the usefulness of a CDSS, and that systems that are used frequently are the most helpful. Measures such as the number of users or frequency of use are difficult to use as criteria because of the infrequent occurrence of cases in clinical practice that are perceived to be diagnostically challenging. Further, the cases for which the system would be used in a live clinical setting are likely to be those that are particularly difficult and may not be the fairest test of the CDSS. While any of the approaches discussed above have problems in being used as the sole gold standard for evaluation of CDSS, researchers, including Ramnarayan et al., Fraser et al., and others who have conducted systematic evaluations of diagnostic CDSS, have appropriately used various combinations of these approaches to provide a multifaceted picture of CDSS performance. However, there is another problem in evaluating CDSS performance that still makes Miller's criterion a challenge for evaluators. Because the output of the CDSS is a combination of the adequacy of the CDSS knowledge base, its inference engine, how the user interacts with the system, and the specific data that are entered, the user can affect the performance of the CDSS. Fraser et al. noted that “giving physicians the flexibility to enter cases in their own fashion…can lead to cases' being entered with insufficient or inaccurate data.”3 My colleagues and I have also found that a decision support system that performed well under ideal conditions when all the case data were entered, performed less well on the same cases when clinicians were free to choose which case data to enter and what system functions to use.8 Also, many of the CDSS are designed to be used interactively and iteratively to provide a variety of perspectives on the patient. If the user does not utilize the system in this interactive fashion (either because of an evaluation design that standardizes the evaluation conditions, or because of lack of time or knowledge of the system capabilities in field studies), the system will perform suboptimally. Furthermore, the influence of the CDSS on the user depends on the user's ability to interpret the CDSS output. The CDSS might suggest the “correct” diagnosis, but the clinicians may not always agree with those suggestions.8 Tsai et al.9 found that nonexpert users of an electrocardiogram (EKG) interpretation system also tended to be influenced by incorrect computer interpretations. These issues are not unique to medical applications. Galletta et al.10 examined the effect of the common word processing “spell checker” and found that under certain circumstances users did worse, not better, when they used the spell-checking software. The interaction of the user and the system in data entry and output interpretation make it especially challenging to address Miller's bottom line criterion. Given the challenges in developing an appropriate gold standard for evaluation of diagnostic CDSS, it may be useful for developers of the next generation of these systems to focus more attention on the intended use of the system as well as on the information presented to the user. Diagnostically challenging cases require reflection over a period of time, examination of the case data from many different perspectives, and rethinking the case as more information becomes available. Cases such as these are memorable precisely because such challenges do not occur frequently. For such cases, a stand-alone, unintegrated CDSS may be fine. However, given that users do not always appropriately recognize diagnostic challenges, a system that can review all patient cases might be preferable, but a standalone system that is designed for extensive and iterative user interaction would be unlikely to be used routinely. Clearly, a CDSS that obtains its input data from an electronic medical record and requires minimal data entry or interaction on the part of the user will be more easily integrated into a busy clinician's workflow. However, even with automated data entry and limited interaction with the CDSS, there is also a cognitive burden in interpreting the output of the CDSS, especially if it produces a lengthy list of diagnostic suggestions, as many of the systems do. Simply truncating the list of suggestions, for the reasons mentioned above, in the discussion of the rank of the correct diagnosis, may not be appropriate. One approach might be to develop a diagnostic CDSS that analyzed the case data to arrive at a differential diagnosis but displayed only for the user a much smaller list of workup or management strategies. Such a system might be easier for the users to process and the researchers to evaluate. If linked to an order entry system, the CDSS might send alerts when a procedure that could rule in or rule out a highly probable diagnosis were omitted. Rather than a lengthy list of possible neurologic diagnoses, for instance, the system would suggest further workup with the neurologic imaging studies, or, if those studies were already ordered, might alert the clinician to consider ordering a vitamin B12 assay if pernicious anemia were also a possibility. Such a system that displayed only general categories of workup or management suggestions, rather than a list of specific diagnoses, might also be more robust in terms of being less sensitive to incomplete or inaccurate data entry. Test cases to evaluate the system would be those in which failure to consider the correct diagnosis is most likely to influence management, rather than those that are diagnostically challenging, selected by the users, or routinely seen, as is typical of most of the studies testing the diagnostic systems. The full differential could be available for the user to review if desired, and further user interaction with the CDSS might also occur. The shorter list of workup/management suggestions would be more likely to be attended to, given clinicians' limited time for interaction with the system. This approach would not negate the importance of diagnostic decision support but would target its performance where it can make the most impact on the users and, ultimately, on the patient.

OBJECTIVE: This study examines how characteristics of clinical cases and physician users relate to the users' perceptions of the usefulness of the Quick Medical Reference (QMR) and their confidence in their diagnoses when supported by the decision support system. METHODS: A national sample (N = 108) of 67 internists, 35 family physicians, and 6 other U.S. physicians used QMR to assist in the diagnosis of written clinical cases. Three sets of eight cases stratified by diagnostic difficulty and the potential of QMR to produce high-quality information were used. A 2 x 2 repeated-measures analysis of variance was used to test whether these factors were associated with perceived usefulness of QMR and physicians' diagnostic confidence after using QMR. Correlations were computed among physician characteristics, ratings of QMR usefulness, and physicians' confidence in their own diagnoses, and between usefulness or confidence and actual diagnostic performance. RESULTS: The analyses showed that QMR was perceived to be significantly more useful (P < 0.05) on difficult cases, on cases where QMR could provide high-quality information, by non-board-certified physicians, and when diagnostic confidence was lower. Diagnostic confidence was higher when comfort with using certain QMR functions was higher. The ratings of usefulness or diagnostic confidence were not consistently correlated with diagnostic performance. CONCLUSIONS: The results suggest that users' diagnostic confidence and perceptions of QMR usefulness may be associated more with their need for decision support than with their actual diagnostic performance when using the system. Evaluators may fail to find a diagnostic decision support system useful if only easy cases are tested, if correct diagnoses are not in the system's knowledge base, or when only highly trained physicians use the system.

PURPOSE: This study examines how the information provided by a diagnostic decision support system for clinical cases of varying diagnostic difficulty affects physicians' diagnostic performance. METHODS: A national sample of 67 internists, 35 family physicians, and 6 other physicians used the Quick Medical Reference (QMR) diagnostic decision support system to assist them in the diagnosis of written clinical cases. Three sets of eight cases, stratified by diagnostic difficulty and the potential of QMR to produce high-quality information, were used. The effects of using QMR on three measures of physicians' diagnostic performance were analyzed using analyses of variance. RESULTS: Physicians' diagnostic performance was significantly higher (p < 0.01) on the easier cases and the cases for which QMR could provide higher-quality information. CONCLUSIONS: Physicians' diagnostic performance can be strongly influenced by the quality of information the system produces and the type of cases on which the system is used.

OBJECTIVE: To examine the relationships among different performance scores for each of four diagnostic decision support systems (DDSSs). DESIGN: Intercorrelations among seven performance scores on a set of 105 cases for each of four DDSSs (DXplain, Iliad, Meditel, QMR) were computed. METHODS: The performance scores for each case reflected: 1) presence or absence of the case diagnosis in the DDSS knowledge base; 2) presence or absence of the correct diagnosis anywhere on the DDSS diagnosis list; 3) presence or absence of the correct diagnosis in the top ten diagnoses; 4) relevance of the DDSS diagnosis list; 5) comprehensiveness of the DDSS diagnosis list; 6) whether the DDSS suggested additional diagnoses to the experts' list; and 7) the length of the DDSS diagnosis list. RESULTS: For all DDSSs, the two Correct Diagnosis scores (top ten and total list) were significantly related: 1) to the presence of the correct diagnosis in the knowledge base; 2) to the Comprehensiveness score; and 3) to each other. There were significant differences among the four DDSSs on the magnitude and/or direction of the relationships between: 1) the two Correct Diagnosis scores; 2) the Relevance and Length scores; and 3) the Relevance and Additional Diagnoses scores. CONCLUSION: The production of a correct diagnosis for a given case is not related to the number of diagnoses suggested by the DDSS and, across different DDSSs, is not consistently related to other measures of performance. These data indicate that multiple measures are needed to fully describe the performance of a DDSS.