Artificial Intelligence (AI) holds significant promise for healthcare but often struggles to transition from development to clinical integration. This paper argues that Learning Health Systems (LHS)-socio-technical ecosystems designed for continuous data-driven improvement-provide a potential "glide path" for safe, sustainable AI deployment. Just as modern aviation depends on instrument landing systems, the safe and effective integration of AI into healthcare requires the socio-technical infrastructure of LHSs, that enable iterative development and monitoring of AI tools, integrating clinical, technical, and ethical considerations through stakeholder collaboration. They address key challenges in AI implementation, including model generalizability, workflow integration, and transparency, by embedding co-creation, real-world evaluation, and continuous learning into care processes. Unlike static deployments, LHSs support the dynamic evolution of AI systems, incorporating feedback and recalibration to mitigate performance drift and bias. Moreover, they embed governance and regulatory functions-clarifying accountability, supporting data and model provenance, and upholding FAIR (Findable, Accessible, Interoperable, Reusable) principles. LHSs also promote "human-in-the-loop" safety through structured studies of human-AI interaction and shared decision-making. The paper outlines practical steps to align AI with LHS frameworks, including investment in data infrastructure, continuous model monitoring, and fostering a learning culture. Embedding AI in LHSs transforms implementation from a one-time event into a sustained, evidence-based learning process that aligns innovation with clinical realities, ultimately advancing patient care, health equity, and system resilience. The arguments build on insights from an international workshop hosted in 2025, offering a strategic vision for the future of AI in healthcare.

OBJECTIVES: There is rapidly growing interest in learning health systems (LHSs) nationally and globally. While the critical role of informatics is recognized, the informatics community has been relatively slow to formalize LHS as a priority area. MATERIALS AND METHODS: We compiled results from a short survey of LHS leaders and American Medical Informatics Association (AMIA) members, discussion from an LHS reception at the AMIA annual meeting, and a follow-up survey to inform priorities at the intersection of LHS and informatics. RESULTS: We present opportunities between informatics and LHS which fell into themes of: Understanding and Context, Shared Resources, Collaboration, Education, Data, Evaluation, and Patient Centeredness. Immediate LHS informatics priorities identified include establishing informatics LHS forum(s), case reports of LHS informatics successes and failures, LHS informatics education resources, and improved understanding of LHS principles in informatics. CONCLUSION: Increased informatics and LHS alignment is critical for advancing this transformative national priority.

Computable biomedical knowledge (CBK) is: "the result of an analytic and/or deliberative process about human health, or affecting human health, that is explicit, and therefore can be represented and reasned upon using logic, formal standards, and mathematical approaches." Representing biomedical knowledge in a machine-interpretable, computable form increases its ability to be discovered, accessed, understood, and deployed. Computable knowledge artifacts can greatly advance the potential for implementation, reproducibility, or extension of the knowledge by users, who may include practitioners, researchers, and learners. Enriching computable knowledge artifacts may help facilitate reuse and translation into practice. Following the examples of 10 Simple Rules papers for scientific code, software, and applications, we present 10 Simple Rules intended to make shared computable knowledge artifacts more useful and reusable. These rules are mainly for researchers and their teams who have decided that sharing their computable knowledge is important, who wish to go beyond simply describing results, algorithms, or models via traditional publication pathways, and who want to both make their research findings more accessible, and to help others use their computable knowledge. These rules are roughly organized into 3 categories: planning, engineering, and documentation. Finally, while many of the following examples are of computable knowledge in biomedical domains, these rules are generalizable to computable knowledge in any research domain.

Dear Editors, We recognize that inefficient and idiosyncratic workflows in health care contribute to a myriad of obstacles for all healthcare stakeholders, including misuse of resources, provider burnout, and increased burden on patients and their caregivers.1 Therefore, we were pleased to read Zayas-Cabán et al’s2 article, “Priorities to accelerate workflow automation in healthcare.” We applaud them for their work illuminating determinants, priorities, and associated strategies for workflow automation. We augment their findings by proposing a seventh priority to stand alongside their original 6—develop and promote infrastructure to facilitate findability, accessibility, interoperability, and reusability of workflows (Table 1). In our view, the automation of healthcare workflows depends on the deployment of reliable, valid, robust, and proven computable biomedical knowledge (CBK) artifacts. We define CBK artifacts as separately packaged software implementations of evidence-based procedural, logical, mathematical, and statistical algorithms.3 We hold this view at an equal level of importance as the need for high-quality, interoperable data and a deep understanding of workflows, as emphasized by Zayas-Cabán et al. Our perspective is that automation of workflows will depend on formalizing and explicitly representing current and desired (optimal) workflows so that they can be tracked and processed by computers in valuable ways. Moreover, to achieve powerful automation, infrastructure must be developed to coordinate and combine computable workflows or process models with AI models, computable guidelines, and other CBK artifacts.

Our goal is to summarize the collective experience of 15 organizations in dealing with uncoordinated efforts that result in unnecessary delays in understanding, predicting, preparing for, containing, and mitigating the COVID-19 pandemic in the US. Response efforts involve the collection and analysis of data corresponding to healthcare organizations, public health departments, socioeconomic indicators, as well as additional signals collected directly from individuals and communities. We focused on electronic health record (EHR) data, since EHRs can be leveraged and scaled to improve clinical care, research, and to inform public health decision-making. We outline the current challenges in the data ecosystem and the technology infrastructure that are relevant to COVID-19, as witnessed in our 15 institutions. The infrastructure includes registries and clinical data networks to support population-level analyses. We propose a specific set of strategic next steps to increase interoperability, overall organization, and efficiencies.

OBJECTIVE: In 2009, a prominent national report stated that 9% of US hospitals had adopted a "basic" electronic health record (EHR) system. This statistic was widely cited and became a memetic anchor point for EHR adoption at the dawn of HITECH. However, its calculation relies on specific treatment of the data; alternative approaches may have led to a different sense of US hospitals' EHR adoption and different subsequent public policy. MATERIALS AND METHODS: We reanalyzed the 2008 American Heart Association Information Technology supplement and complementary sources to produce a range of estimates of EHR adoption. Estimates included the mean and median number of EHR functionalities adopted, figures derived from an item response theory-based approach, and alternative estimates from the published literature. We then plotted an alternative definition of national progress toward hospital EHR adoption from 2008 to 2018. RESULTS: By 2008, 73% of hospitals had begun the transition to an EHR, and the majority of hospitals had adopted at least 6 of the 10 functionalities of a basic system. In the aggregate, national progress toward basic EHR adoption was 58% complete, and, when accounting for measurement error, we estimate that 30% of hospitals may have adopted a basic EHR. DISCUSSION: The approach used to develop the 9% figure resulted in an estimate at the extreme lower bound of what could be derived from the available data and likely did not reflect hospitals' overall progress in EHR adoption. CONCLUSION: The memetic 9% figure shaped nationwide thinking and policy making about EHR adoption; alternative representations of the data may have led to different policy.

OBJECTIVE: To assess measurement practice in clinical decision support evaluation studies. MATERIALS AND METHODS: We identified empirical studies evaluating clinical decision support systems published from 1998 to 2017. We reviewed titles, abstracts, and full paper contents for evidence of attention to measurement validity, reliability, or reuse. We used Friedman and Wyatt's typology to categorize the studies. RESULTS: There were 391 studies that met the inclusion criteria. Study types in this cohort were primarily field user effect studies (n = 210) or problem impact studies (n = 150). Of those, 280 studies (72%) had no evidence of attention to measurement methodology, and 111 (28%) had some evidence with 33 (8%) offering validity evidence; 45 (12%) offering reliability evidence; and 61 (16%) reporting measurement artefact reuse. DISCUSSION: Only 5 studies offered validity assessment within the study. Valid measures were predominantly observed in problem impact studies with the majority of measures being clinical or patient reported outcomes with validity measured elsewhere. CONCLUSION: Measurement methodology is frequently ignored in empirical studies of clinical decision support systems and particularly so in field user effect studies. Authors may in fact be attending to measurement considerations and not reporting this or employing methods of unknown validity and reliability in their studies. In the latter case, reported study results may be biased and effect sizes misleading. We argue that replication studies to strengthen the evidence base require greater attention to measurement practice in health informatics research.

OBJECTIVE: Quantify the variability of patients' problem lists - in terms of the number, type, and ordering of problems - across multiple physicians and assess physicians' criteria for organizing and ranking diagnoses. MATERIALS AND METHODS: In an experimental setting, 32 primary care physicians generated and ordered problem lists for three identical complex internal medicine cases expressed as detailed 2- to 4-page abstracts and subsequently expressed their criteria for ordering items in the list. We studied variability in problem list length. We modified a previously validated rank-based similarity measure, with range of zero to one, to quantify agreement between pairs of lists and calculate a single consensus problem list that maximizes agreement with each physician. Physicians' reasoning for the ordering of the problem lists was recorded. RESULTS: Subjects' problem lists were highly variable. The median problem list length was 8 (range: 3-14) for Case A, 10 (range: 4-20) for Case B, and 7 (range: 3-13) for Case C. The median indices of agreement - taking into account the length, content, and order of lists - over all possible physician pairings was 0.479, 0.371, 0.509, for Cases A, B, and C, respectively. The median agreements between the physicians' lists and the consensus list for each case were 0.683, 0.581, and 0.697 (for Cases A, B, and C, respectively).Out of a possible 1488 pairings, 2 lists were identical. Physicians most frequently ranked problem list items based on their acuity and immediate threat to health. CONCLUSIONS: The problem list is a physician's mental model of a patient's health status. These mental models were found to vary significantly between physicians, raising questions about whether problem lists created by individual physicians can serve their intended purpose to improve care coordination.

OBJECTIVE: The capability to share data, and harness its potential to generate knowledge rapidly and inform decisions, can have transformative effects that improve health. The infrastructure to achieve this goal at scale--marrying technology, process, and policy--is commonly referred to as the Learning Health System (LHS). Achieving an LHS raises numerous scientific challenges. MATERIALS AND METHODS: The National Science Foundation convened an invitational workshop to identify the fundamental scientific and engineering research challenges to achieving a national-scale LHS. The workshop was planned by a 12-member committee and ultimately engaged 45 prominent researchers spanning multiple disciplines over 2 days in Washington, DC on 11-12 April 2013. RESULTS: The workshop participants collectively identified 106 research questions organized around four system-level requirements that a high-functioning LHS must satisfy. The workshop participants also identified a new cross-disciplinary integrative science of cyber-social ecosystems that will be required to address these challenges. CONCLUSIONS: The intellectual merit and potential broad impacts of the innovations that will be driven by investments in an LHS are of great potential significance. The specific research questions that emerged from the workshop, alongside the potential for diverse communities to assemble to address them through a 'new science of learning systems', create an important agenda for informatics and related disciplines.

The term informatics is currently enveloped in chaos. One way to clarify the meaning of informatics is to identify the competencies associated with training in the field, but this approach can conceal the whole that the competencies atomistically describe. This work takes a different approach by offering three higher-level visions of what characterizes the field, viewing informatics as: (1) cross-training between basic informational sciences and an application domain, (2) the relentless pursuit of making people better at what they do, and (3) a field encompassing four related types of activities. Applying these perspectives to describe what informatics is, one can also conclude that informatics is not: tinkering with computers, analysis of large datasets per se, employment in circumscribed health IT workforce roles, the practice of health information management, or anything done using a computer.

This report, based on a workshop jointly sponsored the National Institute of Biomedical Imaging and Biomedical Engineering and the Office of the National Coordinator for Health Information Technology, examines the role and value of images as multimedia data in electronic health records (EHRs). The workshop, attended by a wide range of stakeholders, was motivated in part by the absence of image data from discussions of meaningful use of health information technology. Collectively, the workshop presenters and participants argued that images are not ancillary data and should be central to health information systems to facilitate clinical decisions and higher quality, efficiency, and safety of care. They emphasized that the imaging community has already developed standards that form the basis of interoperability. Despite the apparent value of images, workshop participants also identified challenges and barriers to their implementation within EHRs. Weighing the opportunities and challenges, workshop participants provided their perspectives on possible paths forward toward fully multimedia EHRs.

I thank Dr Hunter for his insightful comments on my paper. I would like to add some observations in response to his. I agree that the “fundamental theorem” as expressed in the paper is silent on method and does not explicate how the person-plus-technology actually becomes better than the person unassisted. The scientific method certainly plays a significant role in this process when the pertinent domain of activity is scientific research. At the same time, the fundamental theorem applies to many domains of work. These domains include research, of course. In that case, the “person” portrayed in the theorem is a scientist. Other pertinent domains are healthcare (in which case the person is a practitioner or a consumer), education (where the person is a student or a teacher), and administration (where the person is a manager). While I agree that the scientific method plays a profound role in making the person better in the domain of research, it is less clear that the scientific method applies directly to the domains of healthcare, education, and administration. For example, in healthcare, a significant body of literature suggests that clinicians do not routinely use hypothetico-deductive reasoning, as described by Dr Hunter. As powerful as this approach can be for the discovery of new knowledge, it is generally inefficient for the application of existing knowledge. Evidence suggests that experienced clinicians use a highly efficient inductive pattern-matching process to arrive at most diagnoses, and employ the scientific method only when the patient's problem does not fit a known pattern. So, I would agree that the scientific method plays a prominent role in making the theorem work but does this primarily in only one of the domains to which the theorem applies. Its role in the other domains is less clear and almost certainly less profound. For that reason, I am much less confident that the scientific method should be included in a general reformulation of the theorem. None. Commissioned; not externally peer reviewed.

This paper proposes, in words and pictures, a "fundamental theorem" to help clarify what informatics is and what it is not. In words, the theorem stipulates that a person working in partnership with an information resource is "better" than that same person unassisted. The theorem is applicable to health care, research, education, and administrative activities. Three corollaries to the theorem illustrate that informatics is more about people than technology; that in order for the theorem to hold, resources must be informative in addition to being correct; and that the theorem can fail to hold for reasons explained by understanding the interaction between the person and the resource.

In 2002-2003, the American College of Medical Informatics (ACMI) undertook a study of the future of informatics training. This project capitalized on the rapidly expanding interest in the role of computation in basic biological research, well characterized in the National Institutes of Health (NIH) Biomedical Information Science and Technology Initiative (BISTI) report. The defining activity of the project was the three-day 2002 Annual Symposium of the College. A committee, comprised of the authors of this report, subsequently carried out activities, including interviews with a broader informatics and biological sciences constituency, collation and categorization of observations, and generation of recommendations. The committee viewed biomedical informatics as an interdisciplinary field, combining basic informational and computational sciences with application domains, including health care, biological research, and education. Consequently, effective training in informatics, viewed from a national perspective, should encompass four key elements: (1). curricula that integrate experiences in the computational sciences and application domains rather than just concatenating them; (2). diversity among trainees, with individualized, interdisciplinary cross-training allowing each trainee to develop key competencies that he or she does not initially possess; (3). direct immersion in research and development activities; and (4). exposure across the wide range of basic informational and computational sciences. Informatics training programs that implement these features, irrespective of their funding sources, will meet and exceed the challenges raised by the BISTI report, and optimally prepare their trainees for careers in a field that continues to evolve.

OBJECTIVE: To identify key features contributing to trainees' development of expertise in microscopic pathology diagnosis, a complex visual task, and to provide new insights to help create computer-based training systems in pathology. DESIGN: Standard methods of information-processing and cognitive science were used to study diagnostic processes (search, perception, reasoning) of 28 novices, intermediates, and experts. Participants examined cases in breast pathology; each case had a previously established gold standard diagnosis. Videotapes correlated the actual visual data examined by participants with their verbal "think-aloud" protocols. MEASUREMENTS: Investigators measured accuracy, difficulty, certainty, protocol process frequencies, error frequencies, and times to key diagnostic events for each case and subject. Analyses of variance, chi-square tests and post-hoc comparisons were performed with subject as the unit of analysis. RESULTS: Level of expertise corresponded with differences in search, perception, and reasoning components of the tasks. Several discrete steps occur on the path to competence, including development of adequate search strategies, rapid and accurate recognition of anatomic location, acquisition of visual data interpretation skills, and transitory reliance on explicit feature identification. CONCLUSION: Results provide the basis for an empirical cognitive model of competence for the complex tasks of microscopic pathology diagnosis. Results will inform the development of computer-based pedagogy tools in this domain

OBJECTIVES: This study sought to assess the ability of medical and nurse practitioner students to use MEDLINE to obtain evidence for answering clinical questions and to identify factors associated with the successful answering of questions. METHODS: A convenience sample of medical and nurse practitioner students was recruited. After completing instruments measuring demographic variables, computer and searching attitudes and experience, and cognitive traits, the subjects were given a brief orientation to MEDLINE searching and the techniques of evidence-based medicine. The subjects were then given 5 questions (from a pool of 20) to answer in two sessions using the Ovid MEDLINE system and the Oregon Health & Science University library collection. Each question was answered using three possible responses that reflected the quality of the evidence. All actions capable of being logged by the Ovid system were captured. Statistical analysis was performed using a model based on generalized estimating equations. The relevance-based measures of recall and precision were measured by defining end queries and having relevance judgments made by physicians who were not associated with the study. RESULTS: Forty-five medical and 21 nurse practitioner students provided usable answers to 324 questions. The rate of correctness increased from 32.3 to 51.6 percent for medical students and from 31.7 to 34.7 percent for nurse practitioner students. Ability to answer questions correctly was most strongly associated with correctness of the answer before searching, user experience with MEDLINE features, the evidence-based medicine question type, and the spatial visualization score. The spatial visualization score showed multi-colinearity with student type (medical vs. nurse practitioner). Medical and nurse practitioner students obtained comparable recall and precision, neither of which was associated with correctness of the answer. CONCLUSIONS: Medical and nurse practitioner students in this study were at best moderately successful at answering clinical questions correctly with the assistance of literature searching. The results confirm the importance of evaluating both search ability and the ability to use the resulting information to accomplish a clinical task.

For those of us who have dedicated our careers to medical informatics, it is easy to feel over-stimulated in the current times. Moore's Law has seemingly been generalized, beyond the hardware we use, to embrace every aspect of our professional lives. Opportunities to apply our science seem to double annually, in ways unforeseeable as recently as five years ago. A field that once was clearly focused on systems to support the care of hospitalized and clinic patients has extended its reach to health information resources for consumers, systems to enhance and protect public health, and systems that support research in genomics and proteomics. We have spawned subfields denoted by prefixes or qualifying phrases, such as “public health informatics,” to mark this trend. Moreover, a field that was solidly rooted in academic medical centers now finds professional representation in for-profit corporations both large and small, in government agencies, and in foundations and professional societies. The field has acquired a distinct entrepreneurial spirit, not at all unwelcome but somehow new and unfamiliar. As our relatively small field engages new problems in new settings, these novel activities are accompanied by an inevitable sense of dilution reflected in specific concerns about our collective future. If we expand our representation into new and diverse environments—as informatics engages Big Science, Big Government, and Big Industry—will there be a sufficient number of us in each of these environments to be influential? Will we retain our own culture or will we dissolve into the cultures of these expanding work settings? Will the information technology deployed in these settings build on the generalizable solutions we have developed and the experience we have accrued, or will these solutions be reinvented? Reflecting this concern, our name—“informatics”—has metamorphosed into something novel and unintended: in many circles “informatics” is coming to mean “anything one does with a computer” in contrast to the more specific research and development connotations that most readers of this journal would attach to the name. So, according to this novel conception, the authors of this manuscript are “doing informatics” as we compose the text using word processing software on our personal computers. It is, indeed, a strange and unsettling time, as many in the field of informatics worry about becoming irrelevant and losing our identity, while the appropriation of our name by most of the rest of the world might suggest that we are more important than ever. Whatever path we take, the future of medical or health informatics is evidently not going to be a straightforward evolution from the past. Under such circumstances, one completely understandable response is a xenophobic circling of the professional wagons, consolidating our identity around a set of very familiar clinically oriented problems as they manifest themselves in academic environments. Another response sees these torsions and exertions as an extraordinary opportunity to take a prominent role in leading biomedicine to wherever it may be headed, wherever that requires us to work. Such concerns are the non-exclusive purview of the elected fellows of the American College of Medical Informatics and established the theme of the College's 2001 Symposium. The stated theme of the symposium was “ACMI.com: The ‘Business' of Informatics in the 21st Century.” This theme, from the outset, was intended to be more metaphoric than literal, above all capturing the spirit of the times in which we live and work. If interpreted literally, the theme captured only a few aspects of the multifaceted changes affecting the field. As discussions occurred during the symposium, scientific, sociologic, and professional issues in the evolution of informatics overtook the business focus implicit in the program title. The symposium's broader points of departure, as stated in the formal call for participation, spanned multiple interrelated trends currently influencing the field: The emergence of “dot coms,” new information technology companies fueled by readily available venture capital that have attracted several prominent researchers from academe to the private sector An increased interest in information resources directly serving the needs of health care consumers The rapid development of the field of bioinformatics, applying information technology to computational problems in modern molecular biology The evolution of mega-systems of health care, and the corresponding emergence of the technologic challenge of integrating information systems across multi-billion-dollar businesses The appearance of a new generation of vendor products, including sophisticated electronic health record systems with intelligent features that had previously been available only to a small number of medical centers that had developed such systems locally The gradual emergence of new business procedures and policies governing health care, requiring care-providing organizations to acquire sophisticated information systems to support administrative and financial aspects of their operations The increasing interest in information systems addressing the health of identified populations, directing the development of information systems to monitor the health states of intact regions, and allowing early intervention in the event of disease outbreaks The symposium activities were designed to acknowledge formally, describe crisply, and analyze insightfully the implications of these sweeping changes, for the College specifically but also for AMIA and the field of medical informatics as a whole. To this end, the symposium's 3 days were organized into a flexible sequence of activities to promote shared understanding of the theme and its importance; to develop a set of more specific topics, not specified in advance, for detailed exploration; and ultimately to allow the participants to explore these more detailed topics and develop conclusions. The nature of the symposium theme led us to embellish the group, traditionally a self-selected set of ACMI fellows, with an experienced person from outside ACMI, to provide additional perspectives. We were fortunate that Dr. Bruce Hochstadt, a principal at Thomas Wiesel Partners, was able to join the 36 ACMI fellows in attendance. Dr. Hochstadt's primary professional concerns are health care information, services, and e-health companies. Dr. Hochstadt was a panelist in the symposium's opening session, contributed to the small-group discussions, and offered a summary perspective at the close of the proceedings. His participation enriched the discussions and lent external validity to the conclusions that emerged from the event. On the initial day of the symposium, an opening panel, moderated by Judy Ozbolt, explored the “changing milieu of informatics.”* Judy's introductory remarks framed the presentations and discussions that followed: What events and developments that we already see will strongly shape the future of health care and medical informatics? How should we respond? What business opportunities hold real promise for bringing the benefits of medical informatics to consumers and clinicians? And what surprise development is lurking over the horizon that may revolutionize again what is possible, what is desirable, and what is necessary? The panelists addressed these questions in the context of an initial set of four themes that had been identified in advance—modern biology, academic medical centers, e-health, and the business of health care. Subsequent deliberations in small groups addressed these same four themes. Following the group sessions, one member from each group prepared a summary on newsprint of the group's findings. These were on display as part of an informal poster session over breakfast the following morning. The second day of the symposium began with an address by Dan Masys, “Beyond the EMR: The Problems Needing Us to Solve.”† Dan's address spawned four themes that spanned and reorganized the ideas introduced on the first day. These themes subsequently became the organizing framework for further discussions and the presentation of the symposium's findings—1) genome-enabled science and health care, 2) education to reduce reliance on memory and opinion, 3) system-mindedness and error reduction, and 4) moving beyond the “guild mentality” in health care and education. Each theme was explored by small groups on the second day, with a poster session to report and share these deliberations on the morning of the symposium's third day. The third day was devoted to further exploration of these themes. We offer as the symposium's central findings an explication of each theme along with some elements of an action agenda for the field of informatics, to include members of AMIA and ACMI. This report's focus on four principal themes, while eliminating many details and side issues that arose during the event, captures the core of what was discussed. The “acmi.com” metaphor became more implicit as the discussions unfolded. Implicit in this exposition is a rejection of wagon-circling in response to the excitement of the current times. Instead we suggest—as an antidote to over-stimulation, differentiation, and dilution—focused attention on a core mission that can be expressed in terms of themes. Such focus can preserve and consolidate the field's identity while promoting substantial research and development to advance the health of the public through direct clinical care, biomedical research, and programs of education. The four themes discussed at the symposium and explicated below are not an exhaustive list, but rather a starting point, and perhaps a rallying-point for a field in serious danger of losing its way. Inevitably, a new and different culture of informatics will emerge as action related to these themes occurs in new organizational contexts and diversified workplaces. Genomic science can be seen in two ways—as a structural component to map the genome and thus understand nature's blueprint for biological action, and, as we increasingly understand how these molecular signatures affect living systems in action, as a functional component. With the structural map of many organisms now complete, we are moving beyond a primary focus on molecular sequence and well into the era of functional genomics, which now makes it possible to identify molecular signatures of specific diseases. Microarray technology to measure gene expression, the primary scientific apparatus of functional genomics, generates data that are multi-dimensional and noisy. Current analytic methodology, using statistical clustering algorithms and manual identification of gene loci, is crude but nonetheless has identified some important linkages between the approximately 10,000 known human genes, with 20,000 known expression patterns, and a small number of diseases such as large cell lymphoma for which gene expression is strongly predictive of prognosis.1 A next generation of analytic tools for functional genomics envisions the simultaneous measurement of the expression patterns of all the approximately 35,000 genes that the human genome is expected to comprise. These tools would automatically find correlations between gene expression patterns and normal metabolic homeostasis, nonspecific reactions to disease stimuli, and specific disease states. These correlations, in turn, will enable a next phase of genomics, “personal genomics,” whereby the gene maps of individuals can be used to diagnose risks and states of illness and to plan therapy. With a few exceptions, the community of scientists traditionally associated with medical informatics has not played a significant role in the work of structural genomics. This can and should change as we proceed into the eras of functional and personal genomics, because these eras require linkage of molecular data with “person data,” the traditional purview of our field. Moreover, the challenges to create ontologies that promote understanding of the problems to be solved, to build systems that are scalable to the magnitude of the computational challenges that the functional genomics problem creates, to identify new algorithms that offer efficient and creative solutions to the computational problems at hand, and to reconcile the various vocabularies used to represent information are challenges that medical informatics has addressed in the past. The the field of medical informatics, the group identified with AMIA and ACMI, is to on traditional or take to build with the of who are their own to include computational The of not at all is to be many of the most future developments in informatics and health care. between the in the of an a and the external available to that the How an a health care or biomedical a will be by in the but it can be by of in the in the is automatically available to in the world be and it can a reliance on of professional A on in the world will be by the relatively of external information a on in the can be by or by that is not health care, the current problem is on in the leading to reliance by on memory and that have for these the only of medical in the that personal is in it is is one of the of professional in health care. A of medical informatics has been the of resources that the of in the into professional Such tools would the for and of this that can be by external themselves to the of this into of biomedical are now available at the and in the this only one of the How to this external into the and work An external addressing the problem would provide an the the framed the The external be not only and but also and If the has to to that the external is the external will not be an external addressing work should not additional to the of for If the has to work to that the external is the external will not be that the work and problem are but they are becoming more The side of this problem will be as research information and as of electronic health record systems a for integrating these resources into work and and the “guild mentality” addressed in this and more challenges to the of external into professional The culture and of the health in the of clinical who about a or the or of who a are in the who and a are by and the biomedical is as the of the for using this external is to and rather than to it education programs that a with with the that these will be both and these These into the work of in the and in and by the next generation of it is a relatively the is the most of promoting the of external into and should be for their to find and external information with their personal This is the primary challenge medical A field that has health education now has the opportunity to revolutionize the to To a in all of can be by systems that it for to the and easy for to the that in from to this report medical at the more above the leading of of more experience medical each in the by medical to the of health care the its second the A for the 21st the on of in that is perhaps most is the of real health care systems to address both and or applying in information technology to administrative and clinical medical are not because information that the information that is not available and it is The challenge to ACMI, and the field of biomedical informatics is We the and of and information into new and systems of health care As we to this problems of technology and information with problems of professional culture and care to on the and of to problems that the of human If we are to reduce we our culture to one in which their human systems that for their inevitable human that requires that we first build the systems and their can we to these new tools into their and to How can the field of medical informatics promote rapid development of tools to the culture and systems of health a few academic medical centers, have already developed and systems for care and that provide both support and from clinical data for These should and to The on error of and care on current scientific and to the The additional on error of support systems that both data and The additional on error of systems that data to for care and that provide specific support The of to the changes in and of care that the information systems The changes in organizational that health care systems is, and information from care data and current to and care and The and issues that for is not as the of the but as a shared in a of care A using research and data would the of data across and and aspects of and systems of have to be in ways to their own and of these and should be the of to generalizable from such findings detailed for using information systems to reduce medical and at each from to by an the on of in the of the and the validity of the findings. such a of the most information systems designed to reduce medical will require a significant of Informatics and would have to this of the of the science a would provide the of allowing to while how to to the If a change were for health care to information resources to and information from the into their clinical of would be to a that health professional as an from and more to its own and than to as a whole. the requires care and all to health as in an to a shared of public health and The of health care by of would to an of by and and of and a the for health and the to in that and between health would be as became the for The of the work would to changes in would be directly into and careers would be and would be used to monitor science would be it was most A would be between the and their The would be to in a focused would have a For all those moving beyond the will not be The group that addressed this during the ACMI symposium that the is and is, the for each health is the professional in the are and that the real was an care beyond the was more or than a to the The group identified related to each aspect of the and that would be to the For to of the the would require what the of health care and from professional health care and the and by and the A new would have to be to the of health care and of and in health care. in the new would take for their own health, it is to explore how these individuals information and what of information are The role of genomics information in professional care and be The medical informatics community has a role to in health information for by the public and it and Medical also how human and between and patients affect and health The of health care systems provide for and through such will we enable to take for their own health and develop flexible with the The new of health care would thus to human and informatics but with different and of To about would have to the and of to of and health for and through care by and for their the would well to for the in public and how to events and in the and from the public and from would be The and to health at would be an important the has been to an If sufficient interest can be the public and health care it will be to develop for the health care The should describe the various of health care and the public and how these would the work of health care. The address the to each including the public about health through the Such in health care would changes in administrative which also be The the and benefits of the new as well as the for change from the should their on real data from primary and to address the of using informatics to change in health care, in the field of medical informatics should an of the of health professional education and As part of that will develop understanding of the of the new the and the and in the understanding of the concerns of health care and and or how an care can reduce and The four themes from the 2001 ACMI symposium offer an action agenda that across the used traditionally to and sense of the field of These themes emerged from a focused on a field by its own or at the to its name. The themes emerged from the of a field that completely into in the the each is or own and the only informatics that is for each that to These themes a to the to and it with a by which we join around problems in which there is and scientific The themes represent an initial set to us it may be that additional themes are to the and thus the of this different to how we about our field. The of the changes affecting us requires attention and this is seen most clearly in the “acmi.com” theme used to call initial attention to the event. the this was in the of and the symposium the following the of many in health became and it became that and were not going to revolutionize health care or discussions related to e-health became prominent and related to business aspects of informatics more implicit as the symposium unfolded. What of the “acmi.com” theme was an all that the will in the future at as as it has in the and the that we be and to the field of informatics around an action agenda requires a change in the field. for from those who share specific professional to those who share interest in a theme, their professional may The of in which one would be important than what one does in that Informatics would be by its to problems it has and for to these problems as part of its would be by its in The of medical informatics would be to the part we in promoting genome-enabled health care, and the and addressing issues might embellish this The change in the field is in some ways and may not affect what most of us in our professional but it change how we about what we with we the we to our and how and to we

This issue of JAMIA includes two articles submitted in response to a special call, issued on February 29, 2000, for papers reporting “null, negative, or disappointing results.” The motivation for this call was a working hypothesis that medical informatics, like other fields, is afflicted with the academic malady known as publication bias. Publication bias exists when well-executed studies with null results (no significant differences between groups), negative results (favoring the control or placebo arm of the study), or disappointing results (effects that may be positive but of little practical significance) do not find their way into the archival literature. Publication bias therefore skews the archival literature toward work with positive findings. Negative studies have been shown to be 2.6 times less likely than positive studies to reach publication, which creates the potential to distort conclusions drawn from systematic reviews and meta-analyses.1 This phenomenon may have multiple causes—manuscripts may not be recommended for publication by reviewers and editors who see negative or null results as scientifically unimportant, and investigators may not write up such results because they are seen as unlikely to be published—or perhaps embarrassing if they are. Potential embarrassment may be a potent deterrent to informatics researchers, because many investigators also hold administrative positions that might be compromised if these individuals' own studies revealed negative or ambiguous effects of expensive technology that they themselves built or purchased. In response to the call for manuscripts, we received six letters of intent and four completed manuscripts. After review and revision, two manuscripts addressing complementary topics were accepted and appear in this issue. The first, by Patterson and Harasym,2 examines the impact of an educational program for medical students; the second, by Rocha and colleagues,3 explores the impact of a clinical decision support system on clinicians. As the process of soliciting, reviewing, and working with authors to revise these manuscripts unfolded, we began to understand that the seemingly simple concept of a negative study was anything but simple. It is, in fact, a concept laden with subtlety, nuance, and cultural overtones. Using the two published manuscripts as examples, we will examine some of these complexities and the important issues they raise for our field. This exploration begins with our collective value orientation. When we in informatics embark on a study not yet undertaken, or begin reading a newly published study, our values direct us to believe that “the system will work.” We hold this belief because we see the work we do as a fundamental good. In other words, we in informatics are really part scientists, part innovators. As scientists, we try to approach our work dispassionately, believing that just as much can be learned from our results when they are negative as when they are positive. As innovators, we are ideologues who believe that, if we do everything right, our interventions should yield benefits. A well-designed system that is installed impeccably and studied flawlessly is one that, by definition, will yield a beneficial effect. To the extent that we are driven by the values of the innovator, well-executed negative studies should not exist. If a study yields null or negative results, the authors must have done something wrong in designing their information resource, in implementing it, or in conducting the study itself. Further complicating this landscape is the prerogative of the researcher to select the hypotheses or research questions that become the foci of his or her study. It is never possible to address all questions of interest, so this selection process is key to what a study will reveal. When the study results are positive, it is possible that other research questions not explored would have generated negative results; and when results are negative, other questions not explored might have generated positive results. Moreover, once the research questions are selected for a study, researchers can choose among a wide range of methods for addressing these questions. This suggests another source of bias, apart from the review process, that will skew what is reported in the literature. This occurs as researchers, guided implicitly by their values, ask questions and select methods that will cast in a more positive light the systems they study. This practice, to the extent it occurs, has nothing whatsoever to do with scientific fraud or misconduct. It addresses prerogatives that have always been assigned to investigators and how, well within the bounds of accepted conduct, these prerogatives are exercised. Such issues of values and prerogative give more intricate shape to the notion of a “negative study” in informatics. Our initial notion was that negative results could occur only when a well-designed system was deployed impeccably and then examined comprehensively, with results suggesting no beneficial effect. But, as discussed above, this pristine notion of a negative study runs counter to our field's ideology and how investigators, driven by this ideology, may approach their work. No manuscript meeting this strict definition is likely ever to be written or submitted. So for purposes of this issue, we adopted over time a more relaxed definition of a negative study—specifically, a study of an interesting intervention that is well enough conducted to suggest why, for reasons related to system design or implementation or study method, the intervention did not yield the expected beneficial effect. A good negative study is thus a study that contains something significant from which we all can learn. Both “negative reports” included in this issue meet this relaxed definition. The Patterson and Harasym work describes an intervention that is certainly interesting. Their effort exposes medical students to the clinical information systems they will be using routinely in their future practice, while attempting to embed educational experiences into this exposure. We agree with the authors that there have been few, if any, such efforts documented in the archival literature. These authors did an excellent job of integrating their intervention into the students' workflow. There is ample evidence to conclude that these information resources were used by the students, yet no effects on “learning,” as the authors measured learning, were observed. How to measure the outcome of interest is one of many decisions that is a matter of author prerogative. In electing the standard test used in the surgery clerkship, the authors made, in our view, an interesting but perhaps less than optimal choice. This is a conservative choice in the sense that, were statistically significant differences observed, their importance would be unquestioned. But perhaps it was not the best choice, because it may not have been sensitive to the specific effects the authors' intervention engendered. So perhaps a customized test tailored to the outcomes of the intervention would have been better. Were differences observed using a more focused and specific measure, they would have spawned a debate about the significance of these results, but at least there would have been differences to discuss. Interestingly, Patterson and Harasym's conservative choice of outcome measure runs counter to our speculation that informatics researchers will select measures tending to cast their intervention in a positive light. These authors did the opposite. The paper by Rocha and colleagues also satisfies the criterion of having intrinsic interest through the central role envisioned for clinical decision support systems in reducing medical errors and unneeded variations in practice. The authors conclude their paper with multiple reasons why theirs is one of a small number of studies of clinical decision support to yield null results. The study documents, in fact, that their decision support system was not fully integrated into patient care in the clinical environments studied. It relates this experience in a way that helps us understand how difficult the integration process can be, and the care that must be taken to ensure sufficient integration to realize the effects the authors were seeking. Of almost equal interest in this paper is the authors' noble attempt to impose a rigorous experimental method on a dynamic clinical setting, which resulted in a set of matched cases perhaps too small and too narrowly focused on specific clinical problems to detect any effects that may have existed. Like the work of Patterson and Harasym, the paper by Rocha and colleagues is a negative study from which several important lessons can be learned. In conclusion, and with reference to the entire exercise that led to the publication of these two negative studies, we should examine the evidence this experience provides regarding the existence of a significant publication bias in medical informatics. When we issued the call for manuscripts, it was tempting to fantasize that years of pent-up demand would trigger a downpour of papers from authors who had been sitting on interesting, but negative, studies, all this time despairing of a place to publish them. The four manuscripts received were, quantitatively, more of a drizzle than the fantasized torrent. It is possible that our welcoming of negative studies and promises of appropriate review were received cynically, making researchers reluctant to write up data that would report no differences or resubmit work that had perhaps been rejected in the past. Nonetheless, the response to our solicitation seems to lessen the chance that there is a substantial hidden literature depicting a large number of instances where “the system just didn't work” or “no differences were observed.” Beyond that, we enter the realm of speculation as to the extent of publication bias in our field. The motivations to publish and the factors that determine whether a completed work results in a manuscript are themselves complex. Concern about personal or corporate embarrassment may indeed play a major role. As medical system deployments become more extensive and expensive, the consequences attaching to lack of success become Brobdingnagian. Moreover, publication bias does not result exclusively from failure to publish negative studies. In a complementary way, many efforts with positive results do not find their way into the literature because of distraction, overwork, lack of motivation or rewards for publication, or perhaps the desire not to reveal too much about a resource that could be commercializable. These factors also speak to our field's core values. They should be the focus of continuing discussions that go well beyond the issue of negative studies that was our original motivation. We hope that this exercise and the resulting publication of two “negative studies” will, over time, work to reduce whatever level of publication bias exists in medical informatics. We hope we have raised the level of concern about this important scientific issue, and perhaps deepened understanding of what this elusive term means. Authors who have performed careful work should not be hesitant to report this work, even if the results were negative, null, or “disappointing.” We encourage these authors to analyze carefully the reasons these results were obtained, and to be articulate about these factors in their submitted manuscripts.

As we have advanced in medical informatics and created many impressive innovations, we also have learned that technologic developments are not sufficient to bring the value of computer and information technologies to health care systems. This paper proposes a model for improving how we develop and deploy information technology. The authors focus on trends in people, organizational, and social issues (POI/OSI), which are becoming more complex as both health care institutions and information technologies are changing rapidly. They outline key issues and suggest high-priority research areas. One dimension of the model concerns different organizational levels at which informatics applications are used. The other dimension draws on social science disciplines for their approaches to studying implications of POI/OSI in informatics. By drawing on a wide variety of research approaches and asking questions based in social science disciplines, the authors propose a research agenda for high-priority issues, so that the challenges they see ahead for informatics may be met better.

The paper by Hripcsak et al. 1 is a sparkling example of work that is much needed in our field.The preponderance of empirical studies in informatics are demonstration studies concerned with the measured values of a set of variables of interest and what these measured values might say about the efficacy of an information resource.In contrast, Hripcsak et al. have undertaken a measurement study to determine the precision, or reliability, with which these variables of interest can be measured.Why do we need these measurement studies?Science is all about measurement, and a scientist who makes measurements without knowing how well these measurements perform is, after a fashion, flying blind.If medical informatics is going to become a mature science, our measurement methods will have to evolve into a type of reusable technology that researchers can take off the shelf and apply to their study needs.That is to say, we in informatics need measurement methods that can be employed in the same routine way an immunologist does a Western blot assay or a psychologist administers a Minnesota Multiphasic Personality Inventory.This reusable measurement technology will have to be thoroughly pretested and calibrated so that researchers employing the technology know how much error is associated with the measurements they make and, thus, the results they report.Measurement studies, like this one by Hripcsak et al., do exactly that: They determine how much error

In order to determine whether medical students can recognize when an information need has been fulfilled and when it has not, this study examined the quality of medical students' confidence estimates in answering short-answer questions dealing with bacteriology, based upon their personal knowledge alone and what they were able to retrieve from a factual database in microbiology. Twelve students, assessed over three occasions, remained in the final sample. The results indicate that students displayed a positive relationship between their expertise in answering the questions and the amount of overconfidence they indicated (the opposite of the hard–easy effect) for the personal knowledge task using a partial credit format. For the database-assisted task using the partial credit format, students showed less overconfidence in their answers with greater expertise in using the database. For both the personal knowledge and database-assisted tasks using a binary format (all or nothing correct), the students displayed the opposite of the hard–easy effect. We conclude that the patterns of confidence estimates (in the form of Brier scores), and thus students' ability to recognize whether their information need has been fulfilled, differ with varying degrees of expertise in both the personal knowledge responses and the database-assisted responses for both the partial credit and binary formats. Taking into consideration the fact that when subjects, in this case future medical practitioners, are extremely overconfident, they stop looking for information long before they have found material that is relevant, the results have broad implications for medical practice and information seeking.

UNLABELLED: This paper describes details of four scales of a questionnaire-- "Computers in Medical Care"--measuring attributes of computer use, self-reported computer knowledge, computer feature demand, and computer optimism of academic physicians. The reliability (i.e., precision, or degree to which the scale's result is reproducible) and validity (i.e., accuracy, or degree to which the scale actually measures what it is supposed to measure) of each scale were examined by analysis of the responses of 771 full-time academic physicians across four departments at five academic medical centers in the United States. The objectives of this paper were to define the psychometric properties of the scales as the basis for a future demonstration study and, pending the results of further validity studies, to provide the questionnaire and scales to the medical informatics community as a tool for measuring the attitudes of health care providers. METHODOLOGY: The dimensionality of each scale and degree of association of each item with the attribute of interest were determined by principal components factor analysis with orthogonal varimax rotation. Weakly associated items (factor loading < .40) were deleted. The reliability of each resultant scale was computed using Cronbach's alpha coefficient. Content validity was addressed during scale construction; construct validity was examined through factor analysis and by correlational analyses. RESULTS: Attributes of computer use, computer knowledge, and computer optimism were unidimensional, with the corresponding scales having reliabilities of .79, .91, and .86, respectively. The computer-feature demand attribute differentiated into two dimensions: the first reflecting demand for high-level functionality with reliability of .81 and the second demand for usability with reliability of .69. There were significant positive correlations between computer use, computer knowledge, and computer optimism scale scores and respondents' hands-on computer use, computer training, and self-reported computer sophistication. In addition, items posited on the computer knowledge scale to be more difficult generated significantly lower scores. CONCLUSION: The four scales of the questionnaire appear to measure with adequate reliability five attributes of academic physicians' attitudes toward computers in medical care: computer use, self-reported computer knowledge, demand for computer functionality, demand for computer usability, and computer optimism. Results of initial validity studies are positive, but further validation of the scales is needed. The URL of a downloadable HTML copy of the questionnaire is provided.

The continuing development of the field of medical informatics has raised new questions and placed before us new dilemmas. Spurred by the proliferation of information systems to support the broad missions of our institutions, and the evolution of these systems from luxuries to necessities, organizational issues have assumed increasing prominence. Among a dazzling array of organizational issues now before us is the tension between the long-standing academic role of informatics groups within medical centers and the ever-expanding service role. In the academic role, we seek the knowledge to create improved technology and to train the next generation of informatics researchers. In the service role, we seek to put existing technology, developed internally or purchased from vendors, to best use across the full scope of medical center activities. The dilemma before us is not whether both roles are important—the answer to that is clear—but rather how to organize ourselves within our institutions to address both of them. How much organizational distance should exist between the people who carry out these different roles, and who should direct their efforts? Most academic medical centers are actively searching for answers to these organizational questions, and many AMIA members are engaged in this pursuit. The answers obtained will be of profound consequence for our field. The salience of this issue directed its selection as the focus of the ACMI Debate at the closing session of the 1997 AMIA Fall Symposium. The purpose of the debate was not to generate a universal answer, for no such answer exists, but rather to illuminate the many factors that must be considered as our institutions search for an appropriate organizational model. To frame the debate, we intentionally polarized the issue around a specific proposition: Resolved: Academic medical centers should have a single unit responsible for information systems supporting the clinical and academic missions and also should be charged to carry out high-quality education and research in medical informatics. The polarity is such that the affirmative team would argue in favor of one group under one leader who would carry out all roles. The negative team would argue for a significant level of separation. We adapted the standard high school and college debate format to fit the available time and to use competition as a device to promote deeper understanding of key issues. There were no judges and no declared winners. Each team had two members: Warner Slack and William Stead for the affirmative, Mark Frisse and Mark Musen for the negative. The format included eight-minute constructive statements, two-minute cross-examinations, and three-minute closing (rebuttal) statements in this order: First affirmative constructive statement, by Slack Cross-examination of Slack, by Musen First negative constructive statement, by Frisse Cross-examination of Frisse, by Slack Second affirmative constructive statement, by Stead Cross-examination of Stead, by Frisse Second negative constructive statement, by Musen Cross-examination of Musen, by Stead Closing (rebuttal) statements by Frisse, Slack, Musen, and Stead, in that order In preparing this summary, we sought to convey the substance and spirit of the debate in a manner suited to printed text. This narrative follows the order of the debate as it occurred on October 29, 1997, at the AMIA Fall Symposium in Nashville, Tennessee. The constructive statements included here were edited from the notes the debaters used to prepare their statements. The cross-examinations and closing statements were edited from the debate transcripts and retain much of the colloquial language used in the event. We include bibliographic references only to direct quotations and citations used by the debaters themselves. The format of this and any debate, and most notably the polarization of a multifaceted issue, often requires participants to take extreme positions. The debaters' views, in reality, overlap more than this report would suggest. Some of the debaters believed that they could, if asked, argue with equal effectiveness in support of their opponents' position. Statements of the debaters may be at variance with their own personal beliefs and do not represent official policies of the institutions for they the we the 1997 ACMI Academic medical centers in have two with its the one is the medical school with its academic from on on the is the with its from the or on In the are and in the medical as the of and service in the as the in of is the and must the research is in the the academic will with the from time to but as as or in the and the will be and in who are both medical school with their and with their the and the an of is the the and its are to the and their and the The should be a for and the on the the are to the by the the is often In the had the and all often with an that from to the of has and the is in this is more than in clinical The medical school the clinical within their to a The of clinical is both a and a to is for the most the of the In the were in the academic of the but the also a the and how and clinical are for clinical as an academic The was in an academic the but the in the academic their and in the is a different were suited for than for clinical and were the time to the of the in were The or information and or were in this is the in most the clinical if is is to and their often than the systems are of to the and or The to this that have clinical developed their under the of an academic the and of the on the academic clinical and the clinical to the and in the of that the of on to the and with by and in the of In clinical is a medical and in the medical school it in an academic The of an academic of clinical that be with the to do research with in to and systems that for their or to the of systems of that for their academic on that are now systems will their to be responsible for the of the clinical systems used in the of and to the and of clinical in the medical school and and for and academic clinical is to a In summary, as the of in that with the is to the the and the and education the academic clinical and education are the are to as as to and There are for with the academic to clinical of no for with the academic in but not as an academic are the of an academic the medical school the and most of an academic unit such as or clinical to be the of the as a of to in the to and for their to do research both in the as as the In all of these must be in one academic unit for the to be and medical both of for an academic it that an academic unit do a medical would not an an the was the in have any are of to the that is a of a would such as are as academic within the and to to be is an academic that in the of the may be an of are for that on in in the of we all the questions to that of that on with and on our for informatics who to the of information systems in and research the do not the do we from the and they to in this any more than we our from or our from we that our the the is academic as the of we frame our the of debate that have in medical education the of the for medical education before the We our not on the but on the that have to the of the as a and and the of the academic to the of We our on both the high for academic and the more of academic in We these two should and We also frame these different from the of an and from the of an here in the who is engaged in a medical informatics or We these two and us the as we it should In this the is the for a to the in in the that such knowledge will to in technology and will be to the of informatics in the academic medical center is this is by the of in the we is best not by a single unit but by a between an academic unit and a different group the for and support of information The of is a of We many of the for a single unit to be not on on the that this is the only for the research and service to to and to us now to the the organizational has on the of the academic informatics In the should seek an that a to and The is an between the and the one the of academic and in and one will be both for the to and to the This we the academic or is not in in a of and but the to the or the to focus on has In the academic informatics members an between and is by time to and the issues our field. To such an a for the and of information systems in This is not our we take the that of many of the of medical that are as a of and are we that in academic must have a of and In our these one only one should them. one one should train is a most must from To are only they the of a they are not with their who often do not often requires more than one to be us now the issue from a more at the as it is rather than as it should the the are more a and of and many of the medical informatics groups are their systems and by that will not the of but it is will of organizational The these are in is believed to be the best of their but the they from their they if time on do their a to create a research is a in it not that most institutions have a to on the the academic of the clinical and The of our institutions, in often with their in favor of the have with their the by Stead and a it for the academic to in the and do they do of with to the more of information us the from the of a or who is or a in most of us in the preparing for this debate, often of one of our more who how from the to a in the to and on such as these are by and We it is the who these and all or with the from a We who are in this would take on this that most institutions will create a of in both research and institutions in the who have this and it with for more than a that our best informatics research institutions would be of their if they were all information support the institutions that are support for information systems and whether or not these institutions and their are the most for the of that will generate the that will the next generation of We who in to the have at but have on different of us has an focus on medical informatics and most will have an for to of us have from a research to a that have not to a single organizational but to the and between who do the and who in the In summary, we who in to the argue that a single is more than an often to different groups do not We argue that a single of is but it should at the level of our and for a of knowledge information technology on the of responsible for the of the our are to our are to systems and our are engaged in a debate the of academic we in to is it and how a this We that frame the issue this their will be more in with and not in an on the at the for that have in has in clinical to the has in that the and in the within a medical school is not to was to that research and that medical informatics research will be the for clinical not to argue for an a is research in or should the of that research in clinical be to the as it out the have to the is that the is it is for an to the by an in the if we put the research and we are our research at would that two have not research that clinical the in the of a medical a clinical are with the in of this is often the in own as with the the as would be with it is an of The field of medical informatics is at a is We must that we generate that on a systems that or the are and the we will be unit with for and of the and has the best to this Each of these must be but is a and the research a and a of the of the to an that will in research argue that the of research and will be or that will be more be obtained but such are not a of an The from a to support of the of the or from the of a that the and is a but one that must be is no different from how to our missions as we our with their are an a medical a between informatics and the and an to the that it be in is of the In the must use the for or use it before it is for to use it on their In the of the and the be used before the by would be as a by its This is is not a have to to a of informatics and they that they have but that no one would use whether they used in to the they do not a as in clinical is also a to is from informatics. were was of a of a of with a of between a of a to a information systems one of these such as order and to with systems that have placed in such as often To the of and new of that would not be information technology, such as by and it is to out how a new and of information technology would be to the and This be by the informatics research in an in a medical has the and to use a is the on the and the research is no that the of the their clinical systems in an informatics be it is by the that This is for who must in a time The is no for the in technology an if must be to support an be a it is it the it to This of fit requires between responsible for the research and responsible for of has a of in medical with generation and development and to is available for the and is available for the two an to different roles in the and to and between as their the of are in they have a they the of a to it do it to by the of the in the field that was to the informatics is the and knowledge in that to We who both informatics and We who and The that exist will not be the who are responsible for academic informatics with their who are responsible for the and that is much with an organizational In that the was at the of informatics that of the of their clinical occurred on time and and that only of available clinical was used in the that the for is than these the of these statements that the not how to that be in time to our of the that from the to the their will not such an answer from the and of the at We own for the of the we we on it to create a We that level of of our for the of academic medical informatics In do research for do research as as in an that has a that research to be How do by are of a and the at in a if that in the if a that may take to own time is to than it is to are to a if research to support development of that and if do not from the to how to on that have from the for has for to that they are of an unit that also has for one of our informatics to as a unit to an understanding of how the be the was that a not have a that was as a unit was in a to out that the was the role of a unit and that they have such a their were as a and the The to the of the medical informatics research they do not have to that that the of academic medical centers are that academic is on all the of the model. we are on for such as and clinical such as an that informatics to be that the to and to and is by a to the one not a model. is a of team a under is no at do not of as a We are here to academic medical informatics. is not to the of clinical in academic medical are significant and we are of the issues that Stead out in We are to clinical We are we are not here to debate how best to address the of information in academic medical we are here to debate how best to address the of medical informatics as an academic Mark Frisse and that medical informatics has both an academic and a service two are in their In the that the affirmative team a single has for and research in medical informatics and for the of the medical is increasing clinical the academic and the service of medical informatics must more is our that academic research and service have their own and that the and for in one not by any that one in the is to take a and is all in this debate, Warner Slack to the of an academic unit in a school of Warner that to include clinical and Stead as an of an academic the affirmative team these they to the of how academic in medical informatics best their for research and The in this debate has on and on clinical systems in these are for medical they are from the of and research that we all are to the academic would to direct our to the research questions that to be with academic medical informatics. Academic medical informatics in such as medical for and of new technology for information the issue is whether these research questions be by who the of their time to that are best by and by information at own and the academic at has many and academic own medical the of in the of understanding the of members to be of our of the to the in the of to be our new the of to be in our do the of to be for to of we in any of is an service role that to be we are academic questions that to be The people who these two of are not the The for the two of are Stead that the of are all who out service within their and do not to the significant service that have of these of our field a research the was a different was no clinical in academic were no information systems that were available and it was the of who were on the academic to the of on the to with the clinical questions that were at The of the had to address significant service issues in their at the was no one to do is of the of these that is now a and that address the information technology of medical an with of us in would be to to The of the were their at a time the of systems was different from it is In the to the of has we are to the of to how to the of to with the of systems the that in of in the now of are and to There has a new of to with the systems that are now in that in are the to be the systems that we in the clinical of us who are in academic centers not to have with in the the on the affirmative that in the AMIA Fall Symposium is to have in the the of the clinical of The information systems available for in the medical centers are only The not have to on of us in to the will have a key role in that information systems that Warner for clinical will the more that information the of medical informatics research their the more will We are to debate the of and the role of medical informatics in the academic of us in medical informatics to with to the systems that we have and to the systems are used in our of on the have in our in that our a that is in the for or The for this is that we not to our in that our to the medical informatics of our is in medical informatics must with the of the it is much to frame the research questions in that our field to our that of us in are not suited to the of that Stead and Warner Slack would us to now that the field of medical informatics many of our are with the of such that they to have of how our with academic the medical informatics to a of such as the one at how on to research in the on the that information systems have had on and on the in with such How we the to by medical informatics who have in the of How the we do in the of clinical to the of people in the who knowledge and We in medical informatics have to to our but we have not in the of our we to who have both the and the time to that the we are and the we are to and to in medical informatics is it and significant and it the understanding of medical and medical is only as a that our we for people who in a medical at do it is for to a for a for an information that has a significant of it is with to the for to do the is role should academic medical informatics in of within the academic were a and on in the would it to How has the medical informatics research at in How many have that have a in or it is to that the we have has had on the that people have are out that we have had in our in our own but that the broad of the of that we have had in our that the research at is we have a at this and a research has have all but is that they are for order the people the the by order and the these have Mark and Warner are is a position. on personal and would be the to with their the of clinical information systems in this would be the to that of the systems in the will be much In rather that many of the systems we are to will not The with and is that their is not are as if is only and the of a of a and information service for do it all with a not a single that a and do and create an for to do and be a in the research or to many do one and the but be the and for and Warner are that put in of of information systems all We should only be that we would have or of these who a service create a and in an the only is people to do it if the of our medical centers believed in the of informatics as much as we and if we and Warner we would have an generation of technology of to the if this were the academic would not be in its their is in as a for the as any model. it is for a that is the of organizational in a that in the the who is for a that is to a the a different from the one as the for this is to that many service only at the of our in has raised as a of in informatics such are use the in are a and by medical informatics who are now in the but to their academic medical informatics that it is to do medical informatics that if to a the the and do This is not for the but it has its do not have to be at the with a service to it but must and with service would by that if with service and should not be these issues. should an such is the rather than the between research and service are a for but in the is the of these two different of Warner First of the that and much on research is most of us would that research in an academic is in the clinical the and that clinical in an academic as in an academic as would that a with an academic not be to do would argue that within the academic medical center we would the to have an academic To clinical from an academic in to a of the as as to the of medical In would to Mark Frisse and Mark Musen for is an position. Mark In the role of academic in medical we should to the all academic at the of this that academic within medical should not be but should be to the research by believed that the of who new medical to be by who to new rather than by role was to the or to In medical is an for academic their on rather than on academic are not by the of service the is an for a as Warner Slack The are have the time to address broad issues. The from the of a to the questions that to be next and in has a in medical informatics is that often our best people are service roles, the next is to that the field are the of the Academic medical centers in medical informatics who be on that who are not by the of order to were in medical many of the of clinical that were raised in this debate would We would have clinical not would have the to in in our academic medical our academic groups would create such that the would the to our their members of medical school we would be to the who have the in our and that academic We that the of academic in medical informatics should be education and is not to of these that or and systems for the clinical we of medical informatics with more more to our will have and our will be more both in academic centers and in the William This debate on two issues. The is the that to have an have to have one who it is not we are We are an team in people across roles. focus on of the a The issue is the people in academic informatics should have for that their has do not we the for a that a in the that is the only for an academic center to in informatics. at will that they any and they their have a team of and has informatics is a to to take for that or we will not have any role in the The statements of and the participants have us with much to the debate a or had the of the participants to a would be a from this This not the of both the as a and also the different we organize ourselves to do our This debate will in the if the debaters' our organizational focus and institutions the to organize their informatics these to their The of the debate to the participants for their and for the and of their positions. also the ACMI and for its in the of the session and the

This study compared of two modes of access to a biomedical database, in terms of their effectiveness and efficiency in supporting clinical problem solving and in terms of user preferences. Boolean access, which allowed subjects to frame their queries as combinations of keywords, was compared to hypertext access, which allowed subjects to navigate from one database node to another. The accessible biomedical data were identical across system versions. Performance data were collected from two cohorts of first-year medical students, each student randomly assigned to either the Boolean or the hypertext system. Additional attitudinal data were collected from the second cohort. At each of two research sessions (one just before and one just after their bacteriology course), subjects worked eight clinical case problems, first using only their personal knowledge and, subsequently, with aid from the database. Database retrievals enabled students to answer questions they could not answer based on personal knowledge alone. This effect was greater when personal knowledge of bacteriology was lower. There were not statistically significant differences between the two forms of access, in terms of problem-solving effectiveness or efficiency. Students preferred Boolean access over hypertext access.

OBJECTIVES: To assess the effects of incomplete data upon the output of a computerized diagnostic decision support system (DSS), to assess the effects of using the system upon the diagnostic opinions of users, and to explore if these effects vary as a function of clinical experience. DESIGN: Experimental pilot study. Four clusters of nine cases each were constructed and equated for case difficulty. Definitive findings were omitted from the case abstracts. Subjects were randomly assigned to one of four clusters and were trained on the DSS prior to use. SUBJECTS: The study involved 16 physicians at three levels of clinical experience (six general internists, four residents in internal medicine, and six fourth-year medical students), from three academic medical centers. PROCEDURE: Each subject worked up nine cases, first without and then with ILIAD consultation. They were asked to offer up to six potential diagnoses and to list up to three steps that should be the next items in the diagnostic workup. Effects of DSS consultation were measured by changes in the position of the correct diagnosis in the lists of differential diagnoses, pre- and post-consultation. RESULTS: The DSS lists of diagnostic possibilities contained the correct diagnosis in 38% of cases, about midway between the levels of accuracy of residents and attending general internists. In over 70% of cases, the DSS output had no effect on the position of the correct diagnosis in the subjects' lists. The system's diagnostic accuracy was unaffected by the clinical experience of the users.

Charles P. Friedman, PhD, Parvati Dev, PhD; Education and Informatics: it's Time to Join forces, Journal of the American Medical Informatics Association, Volume

OBJECTIVE: Evaluate the performance of a continuous-speech interface to a decision support system. DESIGN: The authors performed a prospective evaluation of a speech interface that matches unconstrained utterances of physicians with controlled-vocabulary terms from Quick Medical Reference (QMR). The performance of the speech interface was assessed in two stages: in the real-time experiment, physician subjects viewed audiovisual stimuli intended to evoke clinical findings, spoke a description of each finding into the speech interface, and then chose from a list generated by the interface the QMR term that most closely matched the finding. Subjects believed that the speech recognizer decoded their utterances; in reality, a hidden experimenter typed utterances into the interface (Wizard-of-Oz experimental design). Later, the authors replayed the same utterances through the speech recognizer and measured how accurately utterances matched with appropriate QMR terms using the results of the real-time experiment as the "gold standard." MEASUREMENTS: The authors measured how accurately the speech-recognition system converted input utterances to text strings (recognition accuracy) and how accurately the speech interface matched input utterances to appropriate QMR terms (semantic accuracy). RESULTS: Overall recognition accuracy was less than 50%. However, using language-processing techniques that match keywords in recognized utterances to keywords in QMR terms, the semantic accuracy of the system was 81%. CONCLUSIONS: Reasonable semantic accuracy was attained when language-processing techniques were used to accommodate for speech misrecognition. In addition, the Wizard-of-Oz experimental design offered many advantages for this evaluation. The authors believe that this technique may be useful to future evaluators of speech-input systems.

Charles P. Friedman, PhD; Where's the Science in Medical Informatics?, Journal of the American Medical Informatics Association, Volume 2, Issue 1, 1 January 199

The relationship between personal knowledge in a domain and searching proficiency in that domain, and the relationship between searching proficiency and database-assisted problem-solving performance were the foci of this study. On four assessment occasions over a 2-year period, 36 medical students solved problems in three biomedical domains (bacteriology, pharmacology, and toxicology) with and without assistance from a factual database in the relevant domain. There was little evidence of any relationship between personal domain knowledge and searching proficiency (i.e., search results, selection of search terms, improvement in selection of search terms over the course of the search, and efficiency). Search results, selection of search terms, and efficiency were found to be related to database-assisted problem-solving performance. © 1995 John Wiley & Sons, Inc.

OBJECTIVE: To assess the degree to which information retrieved from a biomedical database can augment personal knowledge in addressing novel problems, and how the ability to retrieve information evolves over time. DESIGN: This longitudinal study comprised three assessments of two cohorts of medical students. The first assessment occurred just before student course experience in bacteriology, the second occurred just after the course, and the third occurred five months later. At each assessment, the students were initially given a set of bacteriology problems to solve using their personal knowledge only. Each student was then reassigned a sample of problems he or she had answered incorrectly, to work again with assistance from a database containing information about bacteria and bacteriologic concepts. The initial pass through the problems generated a "personal knowledge" score; the second pass generated a "database-assisted" score for each student at each assessment. RESULTS: Over two cohorts, students' personal knowledge scores were very low (approximately 12%) at the first assessment. They rose substantially at the second assessment (approximately 48%) but decreased six months later (approximately 25%). By contrast, database-assisted scores rose linearly: from approximately 44% at the first assessment to approximately 57% at the second assessment, to approximately 75% at the third assessment. CONCLUSION: The persistent increase in database-assisted scores, even when personal knowledge had attenuated, was the most remarkable finding of this study. While some of the increase may be attributed to artifacts of the design, the pattern seems to result from the retained ability to recognize problem-relevant information in a database even when it cannot be recalled.

Careful study of medical informatics research and library-resource projects is necessary to increase the productivity of the research and development enterprise. Medical informatics research projects can present unique problems with respect to evaluation. It is not always possible to adapt directly the evaluation methods that are commonly employed in the natural and social sciences. Problems in evaluating medical informatics projects may be overcome by formulating system development work in terms of a testable hypothesis; subdividing complex projects into modules, each of which can be developed, tested and evaluated rigorously; and utilizing qualitative studies in situations where more definitive quantitative studies are impractical.