OBJECTIVE: Accurately characterizing research trends is critical for identifying cutting-edge scientific breakthroughs in their infancy and informing strategic priorities. This research contributes a pipeline that utilizes generative AI technologies to develop research topic taxonomies from publication keywords and analyze keyword evolution within topics, methodological and domain trends, and topic co-occurrences. We demonstrated the pipeline by conducting a retrospective analysis of biomedical informatics research trends in the Journal of Biomedical Informatics (JBI). METHODS: We identified the JBI publications with keywords available on PubMed, spanning 2011-2025. We downloaded all the keywords and categorized them into methodological innovations and health domains, identified topics, assigned topic names, and constructed their hierarchies, all using large-language models (LLMs). We introduced an automated method for evaluating topics, leveraging MeSH terminology as the underlying knowledge base. RESULTS: Using 6,930 unique keywords from 2,427 publications, we derived 1,028 distinct topics related to methodological innovations, with each topic associated with medians of four keywords (Q1: 2, Q3: 13) and six publications (Q1: 2, Q3: 19). We identified 904 topics related to health domains, with each topic associated with three keywords (Q1: 1, Q3: 11) and four publications (Q1: 1, Q3: 15). Based on the topics, we analyzed the prominent research areas, trends in publication volume, evolution of keyword distributions within each topic, and patterns of co-occurring topics. Among the 2,379 eligible publications, 2,009 (84.4%) exhibited overlap between the keyword-derived MeSH terms and the MeSH terms assigned to the publication by the National Library of Medicine. CONCLUSION: This study presents a method that leverages modern generative AI technologies for retrospective analysis of a scientific field to identify emerging topics and to detect shifts in scholarly focus. Illustrated by data for JBI and correlated with historical background events and policy changes, our findings demonstrate the effectiveness and utility of the methods while providing a powerful lens to understand the evolution of biomedical informatics research priorities in JBI.

OBJECTIVE: IBM(R) Watson for Oncology (WfO) is a clinical decision-support system (CDSS) that provides evidence-informed therapeutic options to cancer-treating clinicians. A panel of experienced oncologists compared CDSS treatment options to treatment decisions made by clinicians to characterize the quality of CDSS therapeutic options and decisions made in practice. METHODS: This study included patients treated between 1/2017 and 7/2018 for breast, colon, lung, and rectal cancers at Bumrungrad International Hospital (BIH), Thailand. Treatments selected by clinicians were paired with therapeutic options presented by the CDSS and coded to mask the origin of options presented. The panel rated the acceptability of each treatment in the pair by consensus, with acceptability defined as compliant with BIH's institutional practices. Descriptive statistics characterized the study population and treatment-decision evaluations by cancer type and stage. RESULTS: Nearly 60% (187) of 313 treatment pairs for breast, lung, colon, and rectal cancers were identical or equally acceptable, with 70% (219) of WfO therapeutic options identical to, or acceptable alternatives to, BIH therapy. In 30% of cases (94), 1 or both treatment options were rated as unacceptable. Of 32 cases where both WfO and BIH options were acceptable, WfO was preferred in 18 cases and BIH in 14 cases. Colorectal cancers exhibited the highest proportion of identical or equally acceptable treatments; stage IV cancers demonstrated the lowest. CONCLUSION: This study demonstrates that a system designed in the US to support, rather than replace, cancer-treating clinicians provides therapeutic options which are generally consistent with recommendations from oncologists outside the US.

A fundamental challenge in the field of clinical decision support is to determine what characteristics of systems make them effective in supporting particular types of clinical decisions. However, we lack such a theory of decision support itself and a model to describe clinical decisions and the systems to support them. This article outlines such a framework. We present a two-stream model of information flow within clinical decision-support systems (CDSSs): reasoning about the patient (the clinical stream), and reasoning about the user (the cognitive-behavioral stream). We propose that CDSS "effectiveness" be measured not only in terms of a system's impact on clinical care, but also in terms of how (and by whom) the system is used, its effect on work processes, and whether it facilitates appropriate decisions by clinicians and patients. Future research into which factors improve the effectiveness of decision support should not regard CDSSs as a single entity, but should instead differentiate systems based on their attributes, users, and the decision being supported.

OBJECTIVE: Given the complexities of the healthcare environment, efforts to develop standardized handoff practices have led to widely varying manifestations of handoff tools. A systematic review of the literature on handoff evaluation studies was performed to investigate the nature, methodological, and theoretical foundations underlying the evaluation of handoff tools and their adequacy and appropriateness in achieving standardization goals. METHOD: We searched multiple databases for articles evaluating handoff tools published between 1 February 1983 and 15 June 2012. The selected articles were categorized along the following dimensions: handoff tool characteristics, standardization initiatives, methodological framework, and theoretical perspectives underlying the evaluation. RESULTS: Thirty-six articles met our inclusion criteria. Handoff evaluations were conducted primarily on electronic tools (64%), with a more recent focus on electronic medical record-integrated tools (36% since 2008). Most evaluations centered on intra-departmental tools (95%). Evaluation studies were quasi-experimental (42%) or observational (50%), with a major focus on handoff-related outcome measures (94%) using predominantly survey-based tools (70%) with user satisfaction metrics (53%). Most of the studies (81%) based their evaluation on aspects of standardization that included continuity of care and patient safety. CONCLUSIONS: The nature, methodological, and theoretical foundations of handoff tool evaluations varied significantly in terms of their quality and rigor, thereby limiting their ability to inform strategic standardization initiatives. Future research should utilize rigorous, multi-method qualitative and quantitative approaches that capture the contextual nuances of handoffs, and evaluate their effect on patient-related outcomes.

OBJECTIVE: To present a framework for combining implicit knowledge acquisition from multiple experts with machine learning and to evaluate this framework in the context of anemia alerts. MATERIALS AND METHODS: Five internal medicine residents reviewed 18 anemia alerts, while 'talking aloud'. They identified features that were reviewed by two or more physicians to determine appropriate alert level, etiology and treatment recommendation. Based on these features, data were extracted from 100 randomly-selected anemia cases for a training set and an additional 82 cases for a test set. Two staff internists assigned an alert level, etiology and treatment recommendation before and after reviewing the entire electronic medical record. The training set of 118 cases (100 plus 18) and the test set of 82 cases were explored using RIDOR and JRip algorithms. RESULTS: The feature set was sufficient to assess 93% of anemia cases (intraclass correlation for alert level before and after review of the records by internists 1 and 2 were 0.92 and 0.95, respectively). High-precision classifiers were constructed to identify low-level alerts (precision p=0.87, recall R=0.4), iron deficiency (p=1.0, R=0.73), and anemia associated with kidney disease (p=0.87, R=0.77). DISCUSSION: It was possible to identify low-level alerts and several conditions commonly associated with chronic anemia. This approach may reduce the number of clinically unimportant alerts. The study was limited to anemia alerts. Furthermore, clinicians were aware of the study hypotheses potentially biasing their evaluation. CONCLUSION: Implicit knowledge acquisition, collaborative filtering and machine learning were combined automatically to induce clinically meaningful and precise decision rules.

Federal legislation (Health Information Technology for Economic and Clinical Health (HITECH) Act) has provided funds to support an unprecedented increase in health information technology (HIT) adoption for healthcare provider organizations and professionals throughout the U.S. While recognizing the promise that widespread HIT adoption and meaningful use can bring to efforts to improve the quality, safety, and efficiency of healthcare, the American Medical Informatics Association devoted its 2009 Annual Health Policy Meeting to consideration of unanticipated consequences that could result with the increased implementation of HIT. Conference participants focused on possible unintended and unanticipated, as well as undesirable, consequences of HIT implementation. They employed an input-output model to guide discussion on occurrence of these consequences in four domains: technical, human/cognitive, organizational, and fiscal/policy and regulation. The authors outline the conference's recommendations: (1) an enhanced research agenda to guide study into the causes, manifestations, and mitigation of unintended consequences resulting from HIT implementations; (2) creation of a framework to promote sharing of HIT implementation experiences and the development of best practices that minimize unintended consequences; and (3) recognition of the key role of the Federal Government in providing leadership and oversight in analyzing the effects of HIT-related implementations and policies.

OBJECTIVE: Contemporary error research suggests that the quest to eradicate error is misguided. Error commission, detection, and recovery are an integral part of cognitive work, even at the expert level. In collaborative workspaces, the perception of potential error is directly observable: workers discuss and respond to perceived violations of accepted practice norms. As perceived violations are captured and corrected preemptively, they do not fit Reason's widely accepted definition of error as "failure to achieve an intended outcome." However, perceived violations suggest the aversion of potential error, and consequently have implications for error prevention. This research aims to identify and describe perceived violations of the boundaries of accepted procedure in a psychiatric emergency department (PED), and how they are resolved in practice. DESIGN: Clinical discourse from fourteen PED patient rounds was audio-recorded. Excerpts from recordings suggesting perceived violations or incidents of miscommunication were extracted and analyzed using qualitative coding methods. The results are interpreted in relation to prior research on vulnerabilities to error in the PED. RESULTS: Thirty incidents of perceived violations or miscommunication are identified and analyzed. Of these, only one medication error was formally reported. Other incidents would not have been detected by a retrospective analysis. CONCLUSIONS: The analysis of perceived violations expands the data available for error analysis beyond occasional reported adverse events. These data are prospective: responses are captured in real time. This analysis supports a set of recommendations to improve the quality of care in the PED and other critical care contexts.

This case study of a serious medication error demonstrates the necessity of a comprehensive methodology for the analysis of failures in interaction between humans and information systems. The authors used a novel approach to analyze a dosing error related to computer-based ordering of potassium chloride (KCl). The method included a chronological reconstruction of events and their interdependencies from provider order entry usage logs, semistructured interviews with involved clinicians, and interface usability inspection of the ordering system. Information collected from all sources was compared and evaluated to understand how the error evolved and propagated through the system. In this case, the error was the product of faults in interaction among human and system agents that methods limited in scope to their distinct analytical domains would not identify. The authors characterized errors in several converging aspects of the drug ordering process: confusing on-screen laboratory results review, system usability difficulties, user training problems, and suboptimal clinical system safeguards that all contributed to a serious dosing error. The results of the authors' analysis were used to formulate specific recommendations for interface layout and functionality modifications, suggest new user alerts, propose changes to user training, and address error-prone steps of the KCl ordering process to reduce the risk of future medication dosing errors.

This report provides an evaluation of the National Library of Medicine-sponsored Woods Hole Medical Informatics (WHMI) course and the extent to which the objectives of the program are achieved. Two studies were conducted to examine the participants' perceptions of both the short-term (spring 2002) and the long-term influences (1993 through 2002) on knowledge, skills, and behavior. Data were collected through the use of questionnaires, semistructured telephone interviews, and participant observation methods to provide both quantitative and qualitative assessment. The participants of the spring 2002 course considered the course to be an excellent opportunity to increase their knowledge and understanding of the field of medical informatics as well as to meet and interact with other professionals in the field to establish future collaborations. Past participants remained highly satisfied with their experience at Woods Hole and its influence on their professional careers and their involvement in a broad range of activities related to medical informatics. This group considered their knowledge and understanding of medical informatics to be of greater quality, had increased their networking with other professionals, and were more confident and motivated to work in the field. Many of the participants feel and show evidence of becoming effective agents of change in their institutions in the area of medical informatics, which is one of the objectives of the program.

InterMed is a collaboration among research groups from Stanford, Harvard, and Columbia Universities. The primary goal of InterMed has been to develop a sharable language that could serve as a standard for modeling computer-interpretable guidelines (CIGs). This language, called GuideLine Interchange Format (GLIF), has been developed in a collaborative manner and in an open process that has welcomed input from the larger community. The goals and experiences of the InterMed project and lessons that the authors have learned may contribute to the work of other researchers who are developing medical knowledge-based tools. The lessons described include (1) a work process for multi-institutional research and development that considers different viewpoints, (2) an evolutionary lifecycle process for developing medical knowledge representation formats, (3) the role of cognitive methodology to evaluate and assist in the evolutionary development process, (4) development of an architecture and (5) design principles for sharable medical knowledge representation formats, and (6) a process for standardization of a CIG modeling language.

Is the solution for medical errors medical or cognitive? In this AMIA2001 panel on medical error, we argued that medical error is primarily an issue for cognitive science and engineering, not for medicine, although the knowledge of the practice of medicine is essential for the research and prevention of medical errors. The three panelists presented studies that demonstrate that cognitive research is the foundation for theories of medical errors and interventions of error reductions.

OBJECTIVE: To explore the use of an observational, cognitive-based approach for differentiating between successful, suboptimal, and failed entry of coded data by clinicians in actual practice, and to detect whether causes for unsuccessful attempts to capture true intended meaning were due to terminology content, terminology representation, or user interface problems. DESIGN: Observational study with videotaping and subsequent coding of data entry events in an outpatient clinic at New York Presbyterian Hospital. PARTICIPANTS: Eight attending physicians, 18 resident physicians, and 1 nurse practitioner, using the Medical Entities Dictionary (MED) to record patient problems, medications, and adverse reactions in an outpatient medical record system. MEASUREMENTS: Classification of data entry events as successful, suboptimal, or failed, and estimation of cause; recording of system response time and total event time. RESULTS: Two hundred thirty-eight data entry events were analyzed; 71.0 percent were successful, 6.3 percent suboptimal, and 22.7 percent failed; unsuccessful entries were due to problems with content in 13.0 percent of events, representation problems in 10.1 percent of events, and usability problems in 5.9 percent of events. Response time averaged 0.74 sec, and total event time averaged 40.4 sec. Of an additional 209 tasks related to drug dose and frequency terms, 94 percent were successful, 0.5 percent were suboptimal, and 6 percent failed, for an overall success rate of 82 percent. CONCLUSIONS: Data entry by clinicians using the outpatient system and the MED was generally successful and efficient. The cognitive-based observational approach permitted detection of false-positive (suboptimal) and false-negative (failed due to user interface) data entry.

As a multidisciplinary field, medical informatics draws on a range of disciplines, such as computer science, information science, and the social and cognitive sciences. The cognitive sciences can provide important insights into the nature of the processes involved in human- computer interaction and help improve the design of medical information systems by providing insight into the roles that knowledge, memory, and strategies play in a variety of cognitive activities. In this paper, the authors survey literature on aspects of medical cognition and provide a set of claims that they consider to be important in medical informatics.

OBJECTIVE: To assess the effects of a computer-based patient record system on human cognition. Computer-based patient record systems can be considered "cognitive artifacts," which shape the way in which health care workers obtain, organize, and reason with knowledge. DESIGN: Study 1 compared physicians' organization of clinical information in paper-based and computer-based patient records in a diabetes clinic. Study 2 extended the first study to include analysis of doctor-patient-computer interactions, which were recorded on video in their entirety. In Study 3, physicians' interactions with computer-based records were followed through interviews and automatic logging of cases entered in the computer-based patient record. RESULTS: Results indicate that exposure to the computer-based patient record was associated with changes in physicians' information gathering and reasoning strategies. Differences were found in the content and organization of information, with paper records having a narrative structure, while the computer-based records were organized into discrete items of information. The differences in knowledge organization had an effect on data gathering strategies, where the nature of doctor-patient dialogue was influenced by the structure of the computer-based patient record system. CONCLUSION: Technology has a profound influence in shaping cognitive behavior, and the potential effects of cognition on technology design needs to be explored.

To the Editor:— We read with great interest the recent article in JAMIA by Hendee, in which he called for the creation of a new NIH institute or center to support biomedical engineering, imaging, and informatics.1 We agree with much of his analysis but wish to offer some additional observations and alternative suggestions for how the informatics community might address the concerns that he has raised. The three cited disciplines clearly do “form the infrastructure on which many of the advances in medical science are built,” and we agree that they “must be nurtured and supported so that they will continue to function as the foundation for the knowledge revolutions of the 21st century.” We question, however, Hendee's claim that “there is no home at the NIH for the basic research that is essential to growth of the intellectual capital of these disciplines” and his characterization of the NIH as composed wholly of disease- or organ-specific research agencies. If one reviews the history of research in bioengineering, imaging, and informatics at the NIH, one will find that two entities have funded a large portion of both the basic and applied work in these fields, neither of which is disease- or organ-specific—the National Center for Research Resources (NRCC, formerly the Division of Research Resources or DRR) and the National Library of Medicine (NLM). Hendee refers to the NLM a few times in the article but fails to capitalize on what we believe is an obvious suggestion: the NLM is the natural home for an expanded intra- and extramural research program in bioengineering, imaging, and informatics. It would be much easier to expand the charge and funding for an existing agency than to create a new one. Furthermore, any new institute or center would, by Hendee's proposal, wrest from the NLM a body of research activities (informatics) that it has long nurtured and promoted, both internally and through its grants program. Furthermore, as Hendee notes, bioengineering, imaging, and informatics are today “experiencing a remarkable convergence spurred by several factors,” which makes even more logical the addition of bioengineering and imaging to the NLM's portfolio of activities. Active coordination or merger with some of the activities of the NCRR might also be appropriate in any such expansion of the NLM and its roles. We also question Hendee's assertion that correction of the current “utilitarian” multi-institute support for bioengineering, imaging, and informatics would ensure that resources “would be invested to support the basic science, engineering, and mathematics essential” to the growth and productivity of the three disciplines. It is unfortunately the case that computing, communications, and engineering have suffered from inadequate basic-research support in recent years, not only at NIH but also across all federal research agencies This was a major finding of the recent report of the Presidential Information Technology Advisory Committee,2 which led to their recommendations for enhanced support for basic research in information technology across all research-related agencies, including NIH. As was noted in the report, one key to accomplishing this at NIH is to push for recognition that “biomedically motivated basic research in information technology should be… viewed both as important information technology research and as fundamental biomedical research.” Until the culture of biomedical research in general, and of the NIH in particular, appreciates the fundamental issues that must be addressed by researchers in bioengineering, imaging, and informatics, we will continue to see the disciplines viewed as what Hendee calls “utilities whose purpose is to produce tools and techniques useful to the research missions of the individual NIH institutes.” Thus, any consolidation of research support for these fields in a reconfigured and expanded NLM will also require both an explicit NLM (and budgetary) commitment to basic research in the fields and a concerted effort to inform leaders and scientists at the other institutes about the fundamental roles that bioengineering, imaging, and informatics play in the evolution of all biomedical science. It is ironic that these proposals for the NIH are being discussed in the United States at precisely the time that a new set of biomedical research entities, the Canadian Institutes of Health Research (CIHR), is being created in Canada. In reviewing the plans for the CIHR (see the CIHR Web site at http://www.cihr.org/), we have been concerned to note that the proposal calls for the creation of multiple institutes that are organ- or disease-oriented, similar to those at the NIH, but without any evident consideration of the need for an entity to support research in bioengineering, imaging, and informatics. We find this troubling, since we are entering an era in which many of the most important breakthroughs are likely to occur at the intersections among traditional disciplines. Since the CIHR is new, it has a wonderful opportunity to create internal entities that will reflect such interdisciplinary opportunities and needs. A proposed CIHR institute that would appear to have an interdisciplinary focus is their Institute of Health Services, Clinical Evaluation and Technology Assessment. This would appear to be similar in motivation to the Agency for Health Care Policy and Research (AHCPR) in the United States, but the title suffers from not even recognizing its research focus explicitly. Altogether missing from the description of this proposed institute are additional, key, cross-cutting areas such as health-sciences educational research (including cognition as it relates to learning and decision making), bioengineering, imaging, and informatics. Thus, the Canadian proposal does not call for the creation of an organization with a charge similar to the intra- and extra-mural research program of the NLM, limited though the latter may be in light of our discussion above. In a letter to the Medical Research Council of Canada, one of us (V.L.P.) has accordingly urged that the CIHR consider creating an institute that would include the mandate of the NLM (but of course without the need to recreate the international library functions of the NLM). Such a new agency could still capture the notions of clinical evaluation and technology assessment but would appropriately broaden its charge to include informatics, bioengineering, and imaging while emphasizing the research, educational, cultural, and social issues that underlie the institute's goals and activities. Thus, the issues addressed in Hendee's article are topical and pertinent for the informatics communities in both the United States and Canada. We suspect that there are analogous issues arising in government research agencies in other countries as well. We accordingly urge a lively dialogue and effective educational and political efforts both in North America and more broadly in the international community.

The InterMed Collaboratory involves five medical institutions (Stanford University, Columbia University, Brigham and Women's Hospital, Massachusetts General Hospital, and McGill University) whose mandate has been to join in the development of shared infrastructural software, tools, and system components that will facilitate and support the development of diverse, institution-specific applications. Collaboration among geographically distributed organizations with different goals and cultures provides significant challenges. One experimental question, underlying all that InterMed has set out to achieve, is whether modern communication technologies can effectively bridge such cultural and geographical gaps, allowing the development of shared visions and cooperative activities so that the end results are greater than any one group could have accomplished on its own. In this paper we summarize the InterMed philosophy and mission, describe our progress over 3 years of collaborative activities, and present study results regarding the nature of the evolving collaborative processes, the perceptions of the participants regarding those processes, and the role that telephone conference calls have played in furthering project goals. Both informal introspection and more formal evaluative work, in which project participants became subjects of study by our evaluation experts from McGill, helped to shift our activities from relatively unfocused to more focused efforts while allowing us to understand the facilitating roles that communications technologies could play in our activities. Our experience and study results suggest that occasional face-to-face meetings are crucial precursors to the effective use of distance communications technologies; that conference calls play an important role in both task-related activities and executive (project management) activities, especially when clarifications are required; and that collaborative productivity is highly dependent upon the gradual development of a shared commitment to a well-defined task that leverages the varying expertise of both local and distant colleagues in the creation of tools of broad utility across the participating sites.

OBJECTIVE: An evaluation of the cognitive processes used in the translation of a clinical guideline from text into an encoded form so that it can be shared among medical institutions. DESIGN: A comparative study at three sites regarding the generation of individual and collaborative representations of a guideline for the management of encephalopathy using the GuideLine Interchange Format (GLIF) developed by members of the InterMed Collaboratory. MEASUREMENTS: Using theories and methods of cognitive science, the study involves a detailed analysis of the cognitive processes used in generating representations in GLIF. The resulting process-outcome measures are used to compare subjects with various types of computer science or clinical expertise and from different institutions. RESULTS: Consistent with prior studies of text comprehension and expertise, the variability in strategies was found to be dependent on the degree of prior experience and knowledge of the domain. Differing both in content and structure, the representations developed by physicians were found to have additional information and organization not explicitly stated in the guidelines, reflecting the physicians' understanding of the underlying pathophysiology. The computer scientists developed more literal representations of the guidelines; addition were mostly limited to specifications mandated by the logic of GLIF itself. Collaboration between physicians and computer scientists resulted in consistent representations that were more than the sum of the separate parts, in that both domain-specific knowledge of medicine and generic knowledge of guideline structure were seamlessly integrated. CONCLUSION: Because of the variable construction of guideline representations, understanding the processes and limitations involved in their generation is important in developing strategies to construct shared representations that are both accurate and efficient. The encoded guidelines developed by teams that include both clinicians and experts in computer-based representations are preferable to those developed by individuals of either type working alone.

Because scientific research is guided by concerns for uncovering "fundamental truths," its time frame differs from that of design, development, and practice, which are driven by immediate needs for practical solutions. In medicine, however, as in other disciplines, basic scientists, developers, and practitioners are being called on increasingly to forge new alliances and work toward common goals. The authors propose that medical informatics be construed as a local science of design. A local science seeks to explain aspects of a domain rather than derive a set of unifying principles. Design is concerned with the creation, implementation, and adaptation of artifacts in a range of settings. The authors explore the implications of this point of view and endeavor to characterize the nature of informatics research, the relationship between theory and practice, and issues of scientific validity and generalizability. They argue for a more pluralistic approach to medical informatics in building a cumulative body of knowledge.

Recent developments in medical informatics research have afforded possibilities for great advances in health care delivery. These exciting opportunities also present formidable challenges to the implementation and integration of technologies in the workplace. As in most domains, there is a gulf between technologic artifacts and end users. Since medical practice is a human endeavor, there is a need for bridging disciplines to enable clinicians to benefit from rapid technologic advances. This is turn necessitates a broadening of disciplinary boundaries to consider cognitive and social factors pertaining to the design and use of technology. The authors argue for a place of prominence for cognitive science. Cognitive science provides a framework for the analysis and modeling of complex human performance and has considerable applicability to a range of issues in informatics. Its methods have been employed to illuminate different facets of design and implementation. This approach has also yielded insights into the mechanisms and processes involved in collaborative design. Cognitive scientific methods and theories are illustrated in the context of two examples that examine human-computer interaction in medical contexts and computer-mediated collaborative processes. The framework outlined in this paper can be used to refine the process of iterative design, end-user training, and productive practice.