OBJECTIVES: We report on findings from a meeting convened by the American College of Medical Informatics (ACMI) to characterize aspects of the patient experience that could be improved using informatics. MATERIALS AND METHODS: The American College of Medical Informatics fellows were invited to share their experiences as patients and suggest informatics approaches that may improve the patient experience. RESULTS: We identified 4 themes: (1) getting the right care, (2) data sharing and data interoperability, (3) guiding low-cost evaluations, and (4) predictive analytics. DISCUSSION: Despite widespread adoption of health IT, patient experiences remain far from optimal. CONCLUSION: The American College of Medical Informatics fellows identified informatics approaches, applications, and research areas that have the potential to improve patient experiences with health care systems.

Eta S. Berner, EdD, FACMI, FIAHSI, died on December 11, 2023, only a month after interacting with many colleagues at the American Medical Informatics Association (AMIA) Annual Symposium in New Orleans. As co-workers, collaborators, and friends we each have personal memories that we hold near and dear. With this editorial, we celebrate her professional contributions. Dr Berner was an author of 20 papers published in Journal of the American Medical Informatics Association (JAMIA),1–20 the most recent in the February 2024 issue. Her 1994 New England Journal of Medicine paper on evaluation of diagnostic decision support systems set the standard for rigorous evaluation of such systems.21 Several of her early JAMIA papers reflected her expertise in decision support,1,4,5 while others represented Dr Berner’s commitment to competency-based informatics education through her own work11,12 as well as her substantial contributions to efforts within AMIA.14 A third area of important contribution published in JAMIA was in the area of professional ethics for the field.8,13,16,17 In addition to her authorship contributions, Dr Berner served multiple terms on the JAMIA Editorial Board providing thoughtful peer-reviews on hundreds of manuscripts.

A panel sponsored by the American College of Medical Informatics (ACMI) at the 2021 AMIA Symposium addressed the provocative question: "Are Electronic Health Records dumbing down clinicians?" After reviewing electronic health record (EHR) development and evolution, the panel discussed how EHR use can impair care delivery. Both suboptimal functionality during EHR use and longer-term effects outside of EHR use can reduce clinicians' efficiencies, reasoning abilities, and knowledge. Panel members explored potential solutions to problems discussed. Progress will require significant engagement from clinician-users, educators, health systems, commercial vendors, regulators, and policy makers. Future EHR systems must become more user-focused and scalable and enable providers to work smarter to deliver improved care.

OBJECTIVE: In response to COVID-19, the informatics community united to aggregate as much clinical data as possible to characterize this new disease and reduce its impact through collaborative analytics. The National COVID Cohort Collaborative (N3C) is now the largest publicly available HIPAA limited dataset in US history with over 6.4 million patients and is a testament to a partnership of over 100 organizations. MATERIALS AND METHODS: We developed a pipeline for ingesting, harmonizing, and centralizing data from 56 contributing data partners using 4 federated Common Data Models. N3C data quality (DQ) review involves both automated and manual procedures. In the process, several DQ heuristics were discovered in our centralized context, both within the pipeline and during downstream project-based analysis. Feedback to the sites led to many local and centralized DQ improvements. RESULTS: Beyond well-recognized DQ findings, we discovered 15 heuristics relating to source Common Data Model conformance, demographics, COVID tests, conditions, encounters, measurements, observations, coding completeness, and fitness for use. Of 56 sites, 37 sites (66%) demonstrated issues through these heuristics. These 37 sites demonstrated improvement after receiving feedback. DISCUSSION: We encountered site-to-site differences in DQ which would have been challenging to discover using federated checks alone. We have demonstrated that centralized DQ benchmarking reveals unique opportunities for DQ improvement that will support improved research analytics locally and in aggregate. CONCLUSION: By combining rapid, continual assessment of DQ with a large volume of multisite data, it is possible to support more nuanced scientific questions with the scale and rigor that they require.

OBJECTIVE: The Genomic Medicine Working Group of the National Advisory Council for Human Genome Research virtually hosted its 13th genomic medicine meeting titled "Developing a Clinical Genomic Informatics Research Agenda". The meeting's goal was to articulate a research strategy to develop Genomics-based Clinical Informatics Tools and Resources (GCIT) to improve the detection, treatment, and reporting of genetic disorders in clinical settings. MATERIALS AND METHODS: Experts from government agencies, the private sector, and academia in genomic medicine and clinical informatics were invited to address the meeting's goals. Invitees were also asked to complete a survey to assess important considerations needed to develop a genomic-based clinical informatics research strategy. RESULTS: Outcomes from the meeting included identifying short-term research needs, such as designing and implementing standards-based interfaces between laboratory information systems and electronic health records, as well as long-term projects, such as identifying and addressing barriers related to the establishment and implementation of genomic data exchange systems that, in turn, the research community could help address. DISCUSSION: Discussions centered on identifying gaps and barriers that impede the use of GCIT in genomic medicine. Emergent themes from the meeting included developing an implementation science framework, defining a value proposition for all stakeholders, fostering engagement with patients and partners to develop applications under patient control, promoting the use of relevant clinical workflows in research, and lowering related barriers to regulatory processes. Another key theme was recognizing pervasive biases in data and information systems, algorithms, access, value, and knowledge repositories and identifying ways to resolve them.

OBJECTIVE: Coronavirus disease 2019 (COVID-19) poses societal challenges that require expeditious data and knowledge sharing. Though organizational clinical data are abundant, these are largely inaccessible to outside researchers. Statistical, machine learning, and causal analyses are most successful with large-scale data beyond what is available in any given organization. Here, we introduce the National COVID Cohort Collaborative (N3C), an open science community focused on analyzing patient-level data from many centers. MATERIALS AND METHODS: The Clinical and Translational Science Award Program and scientific community created N3C to overcome technical, regulatory, policy, and governance barriers to sharing and harmonizing individual-level clinical data. We developed solutions to extract, aggregate, and harmonize data across organizations and data models, and created a secure data enclave to enable efficient, transparent, and reproducible collaborative analytics. RESULTS: Organized in inclusive workstreams, we created legal agreements and governance for organizations and researchers; data extraction scripts to identify and ingest positive, negative, and possible COVID-19 cases; a data quality assurance and harmonization pipeline to create a single harmonized dataset; population of the secure data enclave with data, machine learning, and statistical analytics tools; dissemination mechanisms; and a synthetic data pilot to democratize data access. CONCLUSIONS: The N3C has demonstrated that a multisite collaborative learning health network can overcome barriers to rapidly build a scalable infrastructure incorporating multiorganizational clinical data for COVID-19 analytics. We expect this effort to save lives by enabling rapid collaboration among clinicians, researchers, and data scientists to identify treatments and specialized care and thereby reduce the immediate and long-term impacts of COVID-19.

OBJECTIVE: To develop a collection of concept-relationship-concept tuples to formally represent patients' care context data to inform electronic health record (EHR) development. MATERIALS AND METHODS: We reviewed semantic relationships reported in the literature and developed a manual annotation schema. We used the initial schema to annotate sentences extracted from narrative note sections of cardiology, urology, and ear, nose, and throat (ENT) notes. We audio recorded ENT visits and annotated their parsed transcripts. We combined the results of each annotation into a consolidated set of concept-relationship-concept tuples. We then compared the tuples used within and across the multiple data sources. RESULTS: We annotated a total of 626 sentences. Starting with 8 relationships from the literature, we annotated 182 sentences from 8 inpatient consult notes (initial set of tuples = 43). Next, we annotated 232 sentences from 10 outpatient visit notes (enhanced set of tuples = 75). Then, we annotated 212 sentences from transcripts of 5 outpatient visits (final set of tuples = 82). The tuples from the visit transcripts covered 103 (74%) concepts documented in the notes of their respective visits. There were 20 (24%) tuples used across all data sources, 10 (12%) used only in inpatient notes, 15 (18%) used only in visit notes, and 7 (9%) used only in the visit transcripts. CONCLUSIONS: We produced a robust set of 82 tuples useful to represent patients' care context data. We propose several applications of our tuples to improve EHR navigation, data entry, learning health systems, and decision support.

OBJECTIVE: The study sought to describe the literature describing clinical reasoning ontology (CRO)-based clinical decision support systems (CDSSs) and identify and classify the medical knowledge and reasoning concepts and their properties within these ontologies to guide future research. METHODS: MEDLINE, Scopus, and Google Scholar were searched through January 30, 2019, for studies describing CRO-based CDSSs. Articles that explored the development or application of CROs or terminology were selected. Eligible articles were assessed for quality features of both CDSSs and CROs to determine the current practices. We then compiled concepts and properties used within the articles. RESULTS: We included 38 CRO-based CDSSs for the analysis. Diversity of the purpose and scope of their ontologies was seen, with a variety of knowledge sources were used for ontology development. We found 126 unique medical knowledge concepts, 38 unique reasoning concepts, and 240 unique properties (137 relationships and 103 attributes). Although there is a great diversity among the terms used across CROs, there is a significant overlap based on their descriptions. Only 5 studies described high quality assessment. CONCLUSION: We identified current practices used in CRO development and provided lists of medical knowledge concepts, reasoning concepts, and properties (relationships and attributes) used by CRO-based CDSSs. CRO developers reason that the inclusion of concepts used by clinicians' during medical decision making has the potential to improve CDSS performance. However, at present, few CROs have been used for CDSSs, and high-quality studies describing CROs are sparse. Further research is required in developing high-quality CDSSs based on CROs.

OBJECTIVE: The study sought to describe the literature related to the development of methods for auditing the Unified Medical Language System (UMLS), with particular attention to identifying errors and inconsistencies of attributes of the concepts in the UMLS Metathesaurus. MATERIALS AND METHODS: We applied the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) approach by searching the MEDLINE database and Google Scholar for studies referencing the UMLS and any of several terms related to auditing, error detection, and quality assurance. A qualitative analysis and summarization of articles that met inclusion criteria were performed. RESULTS: Eighty-three studies were reviewed in detail. We first categorized techniques based on various aspects including concepts, concept names, and synonymy (n = 37), semantic type assignments (n = 36), hierarchical relationships (n = 24), lateral relationships (n = 12), ontology enrichment (n = 8), and ontology alignment (n = 18). We also categorized the methods according to their level of automation (ie, automated systematic, automated heuristic, or manual) and the type of knowledge used (ie, intrinsic or extrinsic knowledge). CONCLUSIONS: This study is a comprehensive review of the published methods for auditing the various conceptual aspects of the UMLS. Categorizing the auditing techniques according to the various aspects will enable the curators of the UMLS as well as researchers comprehensive easy access to this wealth of knowledge (eg, for auditing lateral relationships in the UMLS). We also reviewed ontology enrichment and alignment techniques due to their critical use of and impact on the UMLS.

Founded in 1984, the American College of Medical Informatics (ACMI) is a college of elected Fellows from the United States and abroad who have made significant and sustained contributions to the field of biomedical informatics. On November 4, 2018, the 2018 Cohort of Fellows was introduced to the College and attendees at the American Medical Informatics Association (AMIA) Annual Symposium as well as to the public through Tweets. This article includes the introduction for each Fellow in the 2018 Cohort, which was read by Christopher G. Chute (ACMI President), Suzanne Bakken (ACMI Past President), or William M. Tierney (ACMI President-Elect); a somewhat tongue-in-cheek Tweet created by Gretchen Purcell Jackson or James J. Cimino; and a link to their Journal of theAmerican Medical Informatics Association (JAMIA) and JAMIA Open publications. This is followed by the traditional closing remarks of welcome into ACMI. Gregory L. Alexander,PhD, RN, FAAN, FACMI Potter-Brinton Endowed Professor, Sinclair School of Nursing and Department of Health Management and Informatics, University of Missouri

Complaints about electronic health records, including information overload, note bloat, and alert fatigue, are frequent topics of discussion. Despite substantial effort by researchers and industry, complaints continue noting serious adverse effects on patient safety and clinician quality of life. I believe solutions are possible if we can add information to the record that explains the "why" of a patient's care, such as relationships between symptoms, physical findings, diagnostic results, differential diagnoses, therapeutic plans, and goals. While this information may be present in clinical notes, I propose that we modify electronic health records to support explicit representation of this information using formal structure and controlled vocabularies. Such information could foster development of more situation-aware tools for data retrieval and synthesis. Informatics research is needed to understand what should be represented, how to capture it, and how to benefit those providing the information so that their workload is reduced.

Objective: To describe the literature exploring the use of electronic health record (EHR) systems to support creation and use of clinical documentation to guide future research. Materials and Methods: We searched databases including MEDLINE, Scopus, and CINAHL from inception to April 20, 2018, for studies applying qualitative or mixed-methods examining EHR use to support creation and use of clinical documentation. A qualitative synthesis of included studies was undertaken. Results: Twenty-three studies met the inclusion criteria and were reviewed in detail. We briefly reviewed 9 studies that did not meet the inclusion criteria but provided recommendations for EHR design. We identified 4 key themes: purposes of electronic clinical notes, clinicians' reasoning for note-entry and reading/retrieval, clinicians' strategies for note-entry, and clinicians' strategies for note-retrieval/reading. Five studies investigated note purposes and found that although patient care is the primary note purpose, non-clinical purposes have become more common. Clinicians' reasoning studies (n = 3) explored clinicians' judgement about what to document and represented clinicians' thought process in cognitive pathways. Note-entry studies (n = 6) revealed that what clinicians document is affected by EHR interfaces. Lastly, note-retrieval studies (n = 12) found that "assessment and plan" is the most read note section and what clinicians read is affected by external stimuli, care/information goals, and what they know about the patient. Conclusion: Despite the widespread adoption of EHRs, their use to support note-entry and reading/retrieval is still understudied. Further research is needed to investigate approaches to capture and represent clinicians' reasoning and improve note-entry and retrieval/reading.

OBJECTIVE: The study sought to determine availability and use of structured override reasons for drug-drug interaction (DDI) alerts in electronic health records. MATERIALS AND METHODS: We collected data on DDI alerts and override reasons from 10 clinical sites across the United States using a variety of electronic health records. We used a multistage iterative card sort method to categorize the override reasons from all sites and identified best practices. RESULTS: Our methodology established 177 unique override reasons across the 10 sites. The number of coded override reasons at each site ranged from 3 to 100. Many sites offered override reasons not relevant to DDIs. Twelve categories of override reasons were identified. Three categories accounted for 78% of all overrides: "will monitor or take precautions," "not clinically significant," and "benefit outweighs risk." DISCUSSION: We found wide variability in override reasons between sites and many opportunities to improve alerts. Some override reasons were irrelevant to DDIs. Many override reasons attested to a future action (eg, decreasing a dose or ordering monitoring tests), which requires an additional step after the alert is overridden, unless the alert is made actionable. Some override reasons deferred to another party, although override reasons often are not visible to other users. Many override reasons stated that the alert was inaccurate, suggesting that specificity of alerts could be improved. CONCLUSIONS: Organizations should improve the options available to providers who choose to override DDI alerts. DDI alerting systems should be actionable and alerts should be tailored to the patient and drug pairs.

OBJECTIVE: Infobuttons appear as small icons adjacent to electronic health record (EHR) data (e.g., medications, diagnoses, or test results) that, when clicked, access online knowledge resources tailored to the patient, care setting, or task. Infobuttons are required for "Meaningful Use" certification of US EHRs. We sought to evaluate infobuttons' impact on clinical practice and identify features associated with improved outcomes. METHODS: We conducted a systematic review, searching MEDLINE, EMBASE, and other databases from inception to July 6, 2015. We included and cataloged all original research in any language describing implementation of infobuttons or other context-sensitive links. Studies evaluating clinical implementations with outcomes of usage or impact were reviewed in greater detail. Reviewers worked in duplicate to select articles, evaluate quality, and abstract information. RESULTS: Of 599 potential articles, 77 described infobutton implementation. The 17 studies meriting detailed review, including 3 randomized trials, yielded the following findings. Infobutton usage frequency ranged from 0.3 to 7.4 uses per month per potential user. Usage appeared to be influenced by EHR task. Five studies found that infobuttons are used less often than non-context-sensitive links (proportionate usage 0.20-0.34). In 3 studies, users answered their clinical question in > 69% of infobutton sessions. Seven studies evaluated alternative approaches to infobutton design and implementation. No studies isolated the impact of infobuttons on objectively measured patient outcomes. CONCLUSIONS: Weak evidence suggests that infobuttons can help providers answer clinical questions. Research on optimal infobutton design and implementation, and on the impact on patient outcomes and provider behaviors, is needed.

Large amounts of personal health data are being collected and made available through existing and emerging technological media and tools. While use of these data has significant potential to facilitate research, improve quality of care for individuals and populations, and reduce healthcare costs, many policy-related issues must be addressed before their full value can be realized. These include the need for widely agreed-on data stewardship principles and effective approaches to reduce or eliminate data silos and protect patient privacy. AMIA's 2012 Health Policy Meeting brought together healthcare academics, policy makers, and system stakeholders (including representatives of patient groups) to consider these topics and formulate recommendations. A review of a set of Proposed Principles of Health Data Use led to a set of findings and recommendations, including the assertions that the use of health data should be viewed as a public good and that achieving the broad benefits of this use will require understanding and support from patients.

Recently, an important public debate emerged about the digital afterlife of any personal data stored in the cloud. Such debate brings also to attention the importance of transparent management of electronic health record (EHR) data of deceased patients. In this perspective paper, we look at legal and regulatory policies for EHR data post mortem. We analyze observational research situations using EHR data that do not require institutional review board approval. We propose creation of a deceased subject integrated data repository (dsIDR) as an effective tool for piloting certain types of research projects. We highlight several dsIDR challenges in proving death status, informed consent, obtaining data from payers and healthcare providers and the involvement of next of kin.

Point-of-care information needs are common and frequently unmet. One solution to this problem is the use of Infobuttons, which are context-sensitive links from electronic health records (EHR) to knowledge resources, sometimes involving an intermediate broker known as an Infobutton Manager. Health Level Seven (HL7) has developed the Context-Aware Knowledge Retrieval (Infobutton) standard to standardize the integration between EHR systems and knowledge resources. While the standard specifies a set of context attributes and standard terminologies, it leaves to knowledge resources the flexibility to decide how to use these attributes and terminologies to retrieve the most relevant content. This paper describes some of the challenges faced by knowledge resources in trying to locate the most relevant content based on the attribute values for a given Infobutton request. Various approaches to content retrieval are discussed, including the role of indexing with standardized codes, the role of text-based search engines together with their ranking algorithms, and the role of hybrid approaches. Knowledge resource developers must carefully consider business rules, heuristics, and precision/recall tradeoffs when implementing the HL7 Infobutton standard.

OBJECTIVE: The goals of this study were to explore the information needs of case managers who provide services to persons living with HIV (PLWH) and to assess the applicability of the Information Needs Event Taxonomy in a new population. DESIGN: The study design was observational with data collection via an online survey. MEASUREMENTS: Responses to open-ended survey questions about the information needs of case managers (n=94) related to PLWH of three levels of care complexity were categorized using the Information Needs Event Taxonomy. RESULTS: The most frequently identified needs were related to patient education resources (33%), patient data (23%), and referral resources (22%) accounting for 79% of all (N=282) information needs. LIMITATIONS: Study limitations include selection bias, recall bias, and a relatively narrow focus of the study on case-manager information needs in the context of caring for PLWH. CONCLUSION: The study findings contribute to the evidence base regarding information needs in the context of patient interactions by: (1) supporting the applicability of the Information Needs Event Taxonomy and extending it through addition of a new generic question; (2) providing a foundation for the addition of context-specific links to external information resources within information systems; (3) applying a new approach for elicitation of information needs; and (4) expanding the literature regarding addressing information needs in community-based settings for HIV services.

OBJECTIVE: Clinical questions are often long and complex and take many forms. We have built a clinical question answering system named AskHERMES to perform robust semantic analysis on complex clinical questions and output question-focused extractive summaries as answers. DESIGN: This paper describes the system architecture and a preliminary evaluation of AskHERMES, which implements innovative approaches in question analysis, summarization, and answer presentation. Five types of resources were indexed in this system: MEDLINE abstracts, PubMed Central full-text articles, eMedicine documents, clinical guidelines and Wikipedia articles. MEASUREMENT: We compared the AskHERMES system with Google (Google and Google Scholar) and UpToDate and asked physicians to score the three systems by ease of use, quality of answer, time spent, and overall performance. RESULTS: AskHERMES allows physicians to enter a question in a natural way with minimal query formulation and allows physicians to efficiently navigate among all the answer sentences to quickly meet their information needs. In contrast, physicians need to formulate queries to search for information in Google and UpToDate. The development of the AskHERMES system is still at an early stage, and the knowledge resource is limited compared with Google or UpToDate. Nevertheless, the evaluation results show that AskHERMES' performance is comparable to the other systems. In particular, when answering complex clinical questions, it demonstrates the potential to outperform both Google and UpToDate systems. CONCLUSIONS: AskHERMES, available at http://www.AskHERMES.org, has the potential to help physicians practice evidence-based medicine and improve the quality of patient care.

To effectively meet clinician information needs at the point of care, we must understand how their needs are dependent on both context and clinician type. The Infobutton Manager (IM), accessed through a clinical information system, anticipates the clinician's questions and provides links to pertinent electronic resources. We conducted an observational usefulness case study of medical residents (MDs), nurse practitioners (NPs), registered nurses (RNs), and a physician assistant (PA), using the IM in a laboratory setting. Generic question types and success rates for each clinician's information needs were characterized. Question type frequency differed by clinician type. All clinician types asked for institution-specific protocols. The MDs asked about unfamiliar domains, RNs asked about physician order rationales, and NPs asked questions similar to both MDs and RNs. Observational data suggest that IM success rates may be improved by tailoring anticipated questions to clinician type. Clinicians reported that a more visible Infobutton may increase use.

CONTEXT Telemedicine is a promising but largely unproven technology for providing case management services to patients with chronic conditions and lower access to care. OBJECTIVES To examine the effectiveness of a telemedicine intervention to achieve clinical management goals in older, ethnically diverse, medically underserved patients with diabetes. DESIGN, Setting, and Patients A randomized controlled trial was conducted, comparing telemedicine case management to usual care, with blinded outcome evaluation, in 1,665 Medicare recipients with diabetes, aged >/= 55 years, residing in federally designated medically underserved areas of New York State. Interventions Home telemedicine unit with nurse case management versus usual care. Main Outcome Measures The primary endpoints assessed over 5 years of follow-up were hemoglobin A1c (HgbA1c), low density lipoprotein (LDL) cholesterol, and blood pressure levels. RESULTS Intention-to-treat mixed models showed that telemedicine achieved net overall reductions over five years of follow-up in the primary endpoints (HgbA1c, p = 0.001; LDL, p < 0.001; systolic and diastolic blood pressure, p = 0.024; p < 0.001). Estimated differences (95% CI) in year 5 were 0.29 (0.12, 0.46)% for HgbA1c, 3.84 (-0.08, 7.77) mg/dL for LDL cholesterol, and 4.32 (1.93, 6.72) mm Hg for systolic and 2.64 (1.53, 3.74) mm Hg for diastolic blood pressure. There were 176 deaths in the intervention group and 169 in the usual care group (hazard ratio 1.01 [0.82, 1.24]). CONCLUSIONS Telemedicine case management resulted in net improvements in HgbA1c, LDL-cholesterol and blood pressure levels over 5 years in medically underserved Medicare beneficiaries. Mortality was not different between the groups, although power was limited. Trial Registration http://clinicaltrials.gov Identifier: NCT00271739.

OBJECTIVE: To use the semantic and structural properties in the Unified Medical Language System (UMLS) Metathesaurus to characterize and discover potential relationships. DESIGN: The UMLS integrates knowledge from several biomedical terminologies. This knowledge can be used to discover implicit semantic relationships between concepts. In this paper, the authors propose a problem-independent approach for discovering potential terminological relationships that employs semantic abstraction of indirect relationship paths to perform classification and analysis of network theoretical measures such as topological overlap, preferential attachment, graph partitioning, and number of indirect paths. Using different versions of the UMLS, the authors evaluate the proposed approach's ability to predict newly added relationships. MEASUREMENTS: Classification accuracy, precision-recall. RESULTS: Strong discriminative characteristics were observed with a semantic abstraction based classifier (classification accuracy of 91%), the average number of indirect paths, preferential attachment, and graph partitioning to identify potential relationships. The proposed relationship prediction algorithm resulted in 56% recall in top 10 results for new relationships added to subsequent versions of the UMLS between 2005 and 2007. CONCLUSIONS: The UMLS has sufficient knowledge to enable discovery of potential terminological relationships.

The National Cancer Institute (NCI) is developing an integrated biomedical informatics infrastructure, the cancer Biomedical Informatics Grid (caBIG), to support collaboration within the cancer research community. A key part of the caBIG architecture is the establishment of terminology standards for representing data. In order to evaluate the suitability of existing controlled terminologies, the caBIG Vocabulary and Data Elements Workspace (VCDE WS) working group has developed a set of criteria that serve to assess a terminology's structure, content, documentation, and editorial process. This paper describes the evolution of these criteria and the results of their use in evaluating four standard terminologies: the Gene Ontology (GO), the NCI Thesaurus (NCIt), the Common Terminology for Adverse Events (known as CTCAE), and the laboratory portion of the Logical Objects, Identifiers, Names and Codes (LOINC). The resulting caBIG criteria are presented as a matrix that may be applicable to any terminology standardization effort.

OBJECTIVE: Infobuttons are decision support tools that provide links within electronic medical record systems to relevant content in online information resources. The aim of infobuttons is to help clinicians promptly meet their information needs. The objective of this study was to determine whether infobutton links that direct to specific content topics ("topic links") are more effective than links that point to general overview content ("nonspecific links"). DESIGN: Randomized controlled trial with a control and an intervention group. Clinicians in the control group had access to nonspecific links, while those in the intervention group had access to topic links. MEASUREMENTS: Infobutton session duration, number of infobutton sessions, session success rate, and the self-reported impact that the infobutton session produced on decision making. RESULTS: The analysis was performed on 90 subjects and 3,729 infobutton sessions. Subjects in the intervention group spent 17.4% less time seeking for information (35.5 seconds vs. 43 seconds, p = 0.008) than those in the control group. Subjects in the intervention group used infobuttons 20.5% (22 sessions vs. 17.5 sessions, p = 0.21) more often than in the control group, but the difference was not significant. The information seeking success rate was equally high in both groups (89.4% control vs. 87.2% intervention, p = 0.99). Subjects reported a high positive clinical impact (i.e., decision enhancement or knowledge update) in 62% of the sessions. Limitations The exclusion of users with a low frequency of infobutton use and the focus on medication-related information needs may limit the generalization of the results. The session outcomes measurement was based on clinicians' self-assessment and therefore prone to bias. CONCLUSION: The results support the hypothesis that topic links are more efficient than nonspecific links regarding the time seeking for information. It is unclear whether the statistical difference demonstrated will result in a clinically significant impact. However, the overall results confirm previous evidence that infobuttons are effective at helping clinicians to answer questions at the point of care and demonstrate a modest incremental change in the efficiency of information delivery for routine users of this tool.

OBJECTIVES: The UMLS constitutes the largest existing collection of medical terms. However, little has been published about the users and uses of the UMLS. This study sheds light on these issues. DESIGN: We designed a questionnaire consisting of 26 questions and distributed it to the UMLS user mailing list. Participants were assured complete confidentiality of their replies. To further encourage list members to respond, we promised to provide them with early results prior to publication. Sector analysis of the responses, according to employment organizations is used to obtain insights into some responses. RESULTS: We received 70 responses. The study confirms two intended uses of the UMLS: access to source terminologies (75%), and mapping among them (44%). However, most access is just to a few sources, led by SNOMED, MeSH, and ICD. Out of 119 reported purposes of use, terminology research (37), information retrieval (19), and terminology translation (14) lead. Four important observations are that the UMLS is widely used as a terminology (77%), even though it was not designed as one; many users (73%) want the NLM to mark concepts with multiple parents in an indented hierarchy and to derive a terminology from the UMLS (73%). Finally, auditing the UMLS is a top budget priority (35%) for users. CONCLUSIONS: The study reports many uses of the UMLS in a variety of subjects from terminology research to decision support and phenotyping. The study confirms that the UMLS is used to access its source terminologies and to map among them. Two primary concerns of the existing user base are auditing the UMLS and the design of a UMLS-based derived terminology.

OBJECTIVE: The authors aimed to (1) formally represent OASIS-B1 concepts using the Logical Observation Identifiers, Names, and Codes (LOINC) semantic structure; (2) demonstrate integration of OASIS-B1 concepts into a concept-oriented terminology, the Medical Entities Dictionary (MED); (3) examine potential hierarchical structures within LOINC among OASIS-B1 and other nursing terms; and (4) illustrate a Web-based implementation for OASIS-B1 data entry using Dialogix, a software tool with a set of functions that supports complex data entry. DESIGN AND MEASUREMENTS: Two hundred nine OASIS-B1 items were dissected into the six elements of the LOINC semantic structure and then integrated into the MED hierarchy. Each OASIS-B1 term was matched to LOINC-coded nursing terms, Home Health Care Classification, the Omaha System, and the Sign and Symptom Check-List for Persons with HIV, and the extent of the match was judged based on a scale of 0 (no match) to 4 (exact match). OASIS-B1 terms were implemented as a Web-based survey using Dialogix. RESULTS: Of 209 terms, 204 were successfully dissected into the elements of the LOINC semantics structure and integrated into the MED with minor revisions of MED semantics. One hundred fifty-one OASIS-B1 terms were mapped to one or more of the LOINC-coded nursing terms. CONCLUSION: The LOINC semantic structure offers a standard way to add home health care data to a comprehensive patient record to facilitate data sharing for monitoring outcomes across sites and to further terminology management, decision support, and accurate information retrieval for evidence-based practice. The cross-mapping results support the possibility of a hierarchical structure of the OASIS-B1 concepts within nursing terminologies in the LOINC database.

This report provides an evaluation of the National Library of Medicine-sponsored Woods Hole Medical Informatics (WHMI) course and the extent to which the objectives of the program are achieved. Two studies were conducted to examine the participants' perceptions of both the short-term (spring 2002) and the long-term influences (1993 through 2002) on knowledge, skills, and behavior. Data were collected through the use of questionnaires, semistructured telephone interviews, and participant observation methods to provide both quantitative and qualitative assessment. The participants of the spring 2002 course considered the course to be an excellent opportunity to increase their knowledge and understanding of the field of medical informatics as well as to meet and interact with other professionals in the field to establish future collaborations. Past participants remained highly satisfied with their experience at Woods Hole and its influence on their professional careers and their involvement in a broad range of activities related to medical informatics. This group considered their knowledge and understanding of medical informatics to be of greater quality, had increased their networking with other professionals, and were more confident and motivated to work in the field. Many of the participants feel and show evidence of becoming effective agents of change in their institutions in the area of medical informatics, which is one of the objectives of the program.

OBJECTIVE: The Enriched Semantic Network (ESN) was introduced as an extension of the Unified Medical Language System (UMLS) Semantic Network (SN). Its multiple subsumption configuration and concomitant multiple inheritance make the ESN's relationship structures and semantic type assignments different from those of the SN. A technique for deriving the relationship structures of the ESN's semantic types and an automated technique for deriving the ESN's semantic type assignments from those of the SN are presented. DESIGN: The technique to derive the ESN's relationship structures finds all newly inherited relationships in the ESN. All such relationships are audited for semantic validity, and the blocking mechanism is used to block invalid relationships. The mapping technique to derive the ESN's semantic type assignments uses current SN semantic type assignments and preserves nonredundant categorizations, while preventing new redundant categorizations. RESULTS: Among the 426 newly inherited relationships, 326 are deemed valid. Seven blockings are applied to avoid inheritance of the 100 invalid relationships. Sixteen semantic types have different relationship structures in the ESN as compared to those in the SN. The mapping of semantic type assignments from the SN to the ESN avoids the generation of 26,950 redundant categorizations. The resulting ESN contains 138 semantic types, 149 IS-A links, 7,303 relationships, and 1,013,876 semantic type assignments. CONCLUSION: The ESN's multiple inheritance provides more complete relationship structures than in the SN. The ESN's semantic type assignments avoid the existing redundant categorizations appearing in the SN and prevent new ones that might arise due to multiple parents. Compared to the SN, the ESN provides a more accurate unifying semantic abstraction of the UMLS Metathesaurus.

Wireless handheld technology provides new ways to deliver and present information. As with any technology, its unique features must be taken into consideration and its applications designed accordingly. In the clinical setting, availability of needed information can be crucial during the decision-making process. Preliminary studies performed at New York Presbyterian Hospital (NYPH) determined that there are inadequate access to information and ineffective communication among clinicians (potential proximal causes of medical errors). In response to these findings, the authors have been developing extensions to their Web-based clinical information system including PalmCIS, an application that provides access to needed patient information via a wireless personal digital assistant (PDA). The focus was on achieving end-to-end security and developing a highly usable system. This report discusses the motivation behind PalmCIS, design and development of the system, and future directions.

OBJECTIVE: The Unified Medical Language System's (UMLS's) Semantic Network's (SN's) two-tree structure is restrictive because it does not allow a semantic type to be a specialization of several other semantic types. In this article, the SN is expanded into a multiple subsumption structure with a directed acyclic graph (DAG) IS-A hierarchy, allowing a semantic type to have multiple parents. New viable IS-A links are added as warranted. DESIGN: Two methodologies are presented to identify and add new viable IS-A links. The first methodology is based on imposing the characteristic of connectivity on a previously presented partition of the SN. Four transformations are provided to find viable IS-A links in the process of converting the partition's disconnected groups into connected ones. The second methodology identifies new IS-A links through a string matching process involving names and definitions of various semantic types in the SN. A domain expert is needed to review all the results to determine the validity of the new IS-A links. RESULTS: Nineteen new IS-A links are added to the SN, and four new semantic types are also created to support the multiple subsumption framework. The resulting network, called the Enriched Semantic Network (ESN), exhibits a DAG-structured hierarchy. A partition of the ESN containing 19 connected groups is also derived. CONCLUSION: The ESN is an expanded abstraction of the UMLS compared with the original SN. Its multiple subsumption hierarchy can accommodate semantic types with multiple parents. Its representation thus provides direct access to a broader range of subsumption knowledge.

OBJECTIVE: The purposes of the study were (1) to evaluate the usefulness of the International Standards Organization (ISO) Reference Terminology Model for Nursing Diagnoses as a terminology model for defining nursing diagnostic concepts in the Medical Entities Dictionary (MED) and (2) to create the additional hierarchical structures required for integration of nursing diagnostic concepts into the MED. DESIGN AND MEASUREMENTS: The authors dissected nursing diagnostic terms from two source terminologies (Home Health Care Classification and the Omaha System) into the semantic categories of the ISO model. Consistent with the ISO model, they selected Focus and Judgment as required semantic categories for creating intensional definitions of nursing diagnostic concepts in the MED. Because the MED does not include Focus and Judgment hierarchies, the authors developed them to define the nursing diagnostic concepts. RESULTS: The ISO model was sufficient for dissecting the source terminologies into atomic terms. The authors identified 162 unique focus concepts from the 266 nursing diagnosis terms for inclusion in the Focus hierarchy. For the Judgment hierarchy, the authors precoordinated Judgment and Potentiality instead of using Potentiality as a qualifier of Judgment as in the ISO model. Impairment and Alteration were the most frequently occurring judgments. CONCLUSIONS: Nursing care represents a large proportion of health care activities; thus, it is vital that terms used by nurses are integrated into concept-oriented terminologies that provide broad coverage for the domain of health care. This study supports the utility of the ISO Reference Terminology Model for Nursing Diagnoses as a facilitator for the integration process.

We have established that inadequate access to timely information and ineffective communication among patient care team members are frequent events that are proximal causes of medical errors. 1 We are exploring the use of wireless mobile computing technology to help reduce these problems through extensions to the Web-based clinical information system at New York Presbyterian Hospital (NYPH). Currently, there are several wireless devices that allow Web browsing and e-mail messaging along with other features. The advantage of Palm devices over other personal digital assistants (PDA) is size and the enormous amount of software available for the Palm OS. Compared to cellular phones and alphanumeric pagers, the screen size makes them more readable and the input mechanism, more usable. Palm, Inc. is the leading provider of handheld information management solutions, including the Palm V and Palm VII series of connected organizers. 2 Devices in both series are capable of providing wireless service. The clinical information system at NYPH is organized in a way that enables clinical systems on disparate platforms to share patient data. 3 The architecture of the system consists of several layers of which the access layer is pertinent to our wireless extension. This layer carries out the retrieval transactions against the clinical repository, which is the central collection of all patient data at NYPH. These actions are performed by data access modules (DAMs) that were developed for the different transaction types. After studying the different features of the Palm V and Palm VII series, it was decided to utilize the Palm VIIx. With its integrated wireless modem, a Palm VII device along with Palm.Net wireless communication service can provide wireless access to network applications. The Palm VII series introduces web clipping architecture. This involves developing a client-side web clipping application (WCA) to be placed on the Palm device which can be used to interact with a specified content provider's Web site. A combination of Certicom's Elliptic Curve Cryptography (ECC) technology and the standard secure socket layer (SSL) protocol is used to achieve end-to-end encryption of data passed back and forth in web clipping transactions. 4 The goal of the wireless extension to the current clinical information system at NYPH is to provide quick and convenient access to clinical information to health care workers. Development of the application requires implementation of a WCA for the Palm VIIx as well as programs on the server to retrieve data from the clinical repository and respond to queries from the Palm device. The WCA has been constructed using HTML and consists of a simple form asking for a username and password. Upon submission, a CGI (Common Gateway Interface) script on our secure server verifies the user information and displays a list of the patients for that user. The user can then select the patients for whom they need data. Then, upon submission, a program on the secure server utilizing the DAMs will return the requested information. With its wireless capabilities, the Palm VIIx provides a way to view clinical information in a secure, efficient and timely matter. Such access to data can potentially reduce medical errors. Reprinted from the Proceedings of the 2001 AMIA Annual Symposium, with permission.

In order to understand the differing perceptions of information needs and communication patterns of healthcare professionals as they relate to medical errors, we conducted a survey and 5 focus group sessions of inpatient physicians and nurses. Although nurses and physicians stated differing information needs, both groups expressed significant problems with obtaining patient, domain and institution-specific information in a timely manner. Identification of appropriate providers and establishing contact with those people was perceived as the most pressing communication need. All focus group participants felt that communication difficulties were common and could give examples in which such difficulties led to adverse events. Our studies suggest that information needs and communication difficulties are common and can lead to medical errors or near misses. Many of these problems may be amenable to information technology solutions.

Medical errors are common, costly and often preventable. Work in understanding the proximal causes of medical errors demonstrates that systems failures predispose to adverse clinical events. Most of these systems failures are due to lack of appropriate information at the appropriate time during the course of clinical care. Problems with clinical communication are common proximal causes of medical errors. We have begun a project designed to measure the impact of wireless computing on medical errors. We report here on our efforts to develop an ontology representing the intersection of medical errors, information needs and the communication space. We will use this ontology to support the collection, storage and interpretation of project data. The ontology's formal representation of the concepts in this novel domain will help guide the rational deployment of our informatics interventions. A real-life scenario is evaluated using the ontology in order to demonstrate its utility.

OBJECTIVE: Clinical information systems typically present patient data in chronologic order, organized by the source of the information (e.g., laboratory, radiology). This study evaluates the functionality and utility of a knowledge-based system that generates concept-oriented views (organized around clinical concepts such as disease or organ system) of clinical data. DESIGN: The authors have developed a system that uses a knowledge base of interrelationships between medical concepts to infer relationships between data in electronic medical records. They use these inferences to produce summaries, or views, of the data that are relevant to a specific concept of interest. They evaluated the ability of the system to select relevant information, reduce information overload, and support physician information retrieval. MEASUREMENTS: The sensitivity and specificity of the system for identifying relevant patient information were calculated. Effect on information overload was assessed by comparing the amount of information in each view with the amount of information in the entire record. Information retrieval accuracy and cost (time) were used to measure the effect of using concept-oriented views on the efficiency and effectiveness of retrievals. RESULTS: The sensitivity and specificity of the system for identifying relevant clinical information were generally in the range of 70 to 80 percent. Concept-oriented views are effective in reducing the amount of information retrieved (over 80 percent reduction) and, compared with source-oriented views, are able to improve physician retrieval accuracy (p=0.04). CONCLUSION: Computer-generated, concept-oriented views can be used to reduce clinician information overload and improve the accuracy of clinical data retrieval.

OBJECTIVE: To explore the use of an observational, cognitive-based approach for differentiating between successful, suboptimal, and failed entry of coded data by clinicians in actual practice, and to detect whether causes for unsuccessful attempts to capture true intended meaning were due to terminology content, terminology representation, or user interface problems. DESIGN: Observational study with videotaping and subsequent coding of data entry events in an outpatient clinic at New York Presbyterian Hospital. PARTICIPANTS: Eight attending physicians, 18 resident physicians, and 1 nurse practitioner, using the Medical Entities Dictionary (MED) to record patient problems, medications, and adverse reactions in an outpatient medical record system. MEASUREMENTS: Classification of data entry events as successful, suboptimal, or failed, and estimation of cause; recording of system response time and total event time. RESULTS: Two hundred thirty-eight data entry events were analyzed; 71.0 percent were successful, 6.3 percent suboptimal, and 22.7 percent failed; unsuccessful entries were due to problems with content in 13.0 percent of events, representation problems in 10.1 percent of events, and usability problems in 5.9 percent of events. Response time averaged 0.74 sec, and total event time averaged 40.4 sec. Of an additional 209 tasks related to drug dose and frequency terms, 94 percent were successful, 0.5 percent were suboptimal, and 6 percent failed, for an overall success rate of 82 percent. CONCLUSIONS: Data entry by clinicians using the outpatient system and the MED was generally successful and efficient. The cognitive-based observational approach permitted detection of false-positive (suboptimal) and false-negative (failed due to user interface) data entry.

The "vocabulary problem" has long plagued the developers, implementers, and users of computer-based systems. The authors review selected activities of the Health Level 7 (HL7) Vocabulary Technical Committee that are related to vocabulary domain specification for HL7 coded data elements. These activities include: 1) the development of two sets of principles to provide guidance to terminology stakeholders, including organizations seeking to deploy HL7-compliant systems, terminology developers, and terminology integrators; 2) the completion of a survey of terminology developers; 3) the development of a process for HL7 registration of terminologies; and 4) the maintenance of vocabulary domain specification tables. As background, vocabulary domain specification is defined and the relationship between the HL7 Reference Information Model and vocabulary domain specification is described. The activities of the Vocabulary Technical Committee complement the efforts of terminology developers and other stakeholders. These activities are aimed at realizing semantic interoperability in the context of the HL7 Message Development Framework, so that information exchange and use among disparate systems can occur for the delivery and management of direct clinical care as well as for purposes such as clinical research, outcome research, and population health management.

OBJECTIVE: The purpose of this study was to test the adequacy of the Clinical LOINC (Logical Observation Identifiers, Names, and Codes) semantic structure as a terminology model for standardized assessment measures. METHODS: After extension of the definitions, 1, 096 items from 35 standardized assessment instruments were dissected into the elements of the Clinical LOINC semantic structure. An additional coder dissected at least one randomly selected item from each instrument. When multiple scale types occurred in a single instrument, a second coder dissected one randomly selected item representative of each scale type. RESULTS: The results support the adequacy of the Clinical LOINC semantic structure as a terminology model for standardized assessments. Using the revised definitions, the coders were able to dissect into the elements of Clinical LOINC all the standardized assessment items in the sample instruments. Percentage agreement for each element was as follows: component, 100 percent; property, 87.8 percent; timing, 82.9 percent; system/sample, 100 percent; scale, 92.6 percent; and method, 97.6 percent. DISCUSSION: This evaluation was an initial step toward the representation of standardized assessment items in a manner that facilitates data sharing and re-use. Further clarification of the definitions, especially those related to time and property, is required to improve inter-rater reliability and to harmonize the representations with similar items already in LOINC.

Knowledge representation involves enumeration of conceptual symbols and arrangement of these symbols into some meaningful structure. Medical knowledge representation has traditionally focused more on the structure than the symbols. Several significant efforts are under way, at local, national, and international levels, to address the representation of the symbols though the creation of high-quality terminologies that are themselves knowledge based. This paper reviews these efforts, including the Medical Entities Dictionary (MED) in use at Columbia University and the New York Presbyterian Hospital. A decade's experience with the MED is summarized to serve as a proof-of-concept that knowledge-based terminologies can support the use of coded patient data for a variety of knowledge-based activities, including the improved understanding of patient data, the access of information sources relevant to specific patient care problems, the application of expert systems directly to the care of patients, and the discovery of new medical knowledge. The terminological knowledge in the MED has also been used successfully to support clinical application development and maintenance, including that of the MED itself. On the basis of this experience, current efforts to create standard knowledge-based terminologies appear to be justified.

OBJECTIVE: The Unified Medical Language System (UMLS) combines many well-established authoritative medical informatics terminologies in one knowledge representation system. Such a resource is very valuable to the health care community and industry. However, the UMLS is very large and complex and poses serious comprehension problems for users and maintenance personnel. The authors present a representation to support the user's comprehension and navigation of the UMLS. DESIGN: An object-oriented database (OODB) representation is used to represent the two major components of the UMLS-the Metathesaurus and the Semantic Network-as a unified system. The semantic types of the Semantic Network are modeled as semantic type classes. Intersection classes are defined to model concepts of multiple semantic types, which are removed from the semantic type classes. RESULTS: The authors provide examples of how the intersection classes help expose omissions of concepts, highlight errors of semantic type classification, and uncover ambiguities of concepts in the UMLS. The resulting UMLS OODB schema is deeper and more refined than the Semantic Network, since intersection classes are introduced. The Metathesaurus is classified into more mutually exclusive, uniform sets of concepts. The schema improves the user's comprehension and navigation of the Metathesaurus. CONCLUSIONS: The UMLS OODB schema supports the user's comprehension and navigation of the Metathesaurus. It also helps expose and resolve modeling problems in the UMLS.

Controlled medical terminologies (CMTs) are playing central roles in clinical information systems and medical knowledge resource applications. As these terminologies grow, they are able to support more complex tasks but require more intensive efforts to create and maintain them. Several terminologies are evolving into knowledge bases of medical concepts. The knowledge they include is being used to support distributed cognition in two forms: complex medical decisions involving multiple people and applications, and coordination of maintenance of the terminologies themselves.

The InterMed Collaboratory involves five medical institutions (Stanford University, Columbia University, Brigham and Women's Hospital, Massachusetts General Hospital, and McGill University) whose mandate has been to join in the development of shared infrastructural software, tools, and system components that will facilitate and support the development of diverse, institution-specific applications. Collaboration among geographically distributed organizations with different goals and cultures provides significant challenges. One experimental question, underlying all that InterMed has set out to achieve, is whether modern communication technologies can effectively bridge such cultural and geographical gaps, allowing the development of shared visions and cooperative activities so that the end results are greater than any one group could have accomplished on its own. In this paper we summarize the InterMed philosophy and mission, describe our progress over 3 years of collaborative activities, and present study results regarding the nature of the evolving collaborative processes, the perceptions of the participants regarding those processes, and the role that telephone conference calls have played in furthering project goals. Both informal introspection and more formal evaluative work, in which project participants became subjects of study by our evaluation experts from McGill, helped to shift our activities from relatively unfocused to more focused efforts while allowing us to understand the facilitating roles that communications technologies could play in our activities. Our experience and study results suggest that occasional face-to-face meetings are crucial precursors to the effective use of distance communications technologies; that conference calls play an important role in both task-related activities and executive (project management) activities, especially when clarifications are required; and that collaborative productivity is highly dependent upon the gradual development of a shared commitment to a well-defined task that leverages the varying expertise of both local and distant colleagues in the creation of tools of broad utility across the participating sites.

OBJECTIVE: The National Library of Medicine's (NLM) Unified Medical Language System (UMLS) includes a Metathesaurus (Meta), which is a compilation of medical terms drawn from over 30 controlled vocabularies, and a Semantic Net, which contains the semantic types used to categorize Meta concepts and the semantic relations to connect them. Meta has been constructed through lexical matching techniques and human review. The purpose of this study was to audit the Meta using semantic techniques to identify possible inconsistencies. METHODS: Five different techniques were applied: (1) detection of ambiguity in Meta concepts with two or more semantic types, (2) detection of interchangeable keyword synonyms, (3) detection of redundant pairs of Meta concepts (using lexical matching combined with keyword synonyms), (4) detection of inconsistent parent-child relationships in Meta (based on the semantic type information), and (5) discovery of pairs of semantic types for which relations could be added to the Semantic Net, based on "other" relationships between Meta concepts. RESULTS: Of 57,592 concepts with multiple semantic types, 1817 (3.2%) were judged to be ambiguous. Keyword analysis showed 7121 pairs of interchangeable words. Using the keyword pairs, 5031 pairs of potentially redundant concepts were suggested, of which 3274 (65.1%) were judged to actually be redundant. Review of the 100,586 parent-child relationships revealed 544 (0.54%) that were incorrect. Review of the 219,664 "Other" relationships suggested 1299 places in the Semantic Net where relations between pairs of semantic types could be added. CONCLUSION: Semantic techniques, alone or in combination, can be used to audit the UMLS to detect inconsistencies that are not detectable through lexical techniques alone. Use of these methods to augment the UMLS maintenance process will lead to improvement in the UMLS.

As in other areas of society, the Internet and the World Wide Web are becoming important topics in medical informatics. This is evident from the recent American Medical Informatics Association's 1996 Annual Fall Symposium, where the theme was "Beyond the Superhighway: Exploiting the Internet with Medical Informatics." Of the over 330 papers and abstracts published in the Proceedings, one third dealt with the Internet and/or the Web. In some cases, system developers demonstrated how this technology can do old tasks in new ways. In other cases, researchers described new tasks that are now possible with this technology. Still others examined this technology to show how it can be evaluated and improved. This paper summarizes their accomplishments.

OBJECTIVE: To evaluate the performance of tools for authoring patient database queries. DESIGN: Query by Review, a tool that exploits the training that users have undergone to master a result review system, was compared with AccessMed, a vocabulary browser that supports lexical matching and the traversal of hierarchical and semantic links. Seven subjects (Medical Logic Module authors) were asked to use both tools to gather the vocabulary terms necessary to perform each of eight laboratory queries. MEASUREMENTS: The proportion of queries that were correct; intersubject agreement. RESULTS: Query by Review had better performance than AccessMed (38% correct queries versus 18%, p = 0.002), but both figures were low. Poor intersubject agreement (28% for Query by Review and 21% for AccessMed) corroborated the relatively low performance. Subjects appeared to have trouble distinguishing laboratory tests from laboratory batteries, picking terms relevant to the particular data type required, and using classes in the vocabulary's hierarchy. CONCLUSION: Query by Review, with its more constrained user interface, performed somewhat better than AccessMed, a more general tool. Neither tool achieved adequate performance, however, which points to the difficulty of formulating a query for a clinical database and the need for further work.

Clinical computing application development at Columbia-Presbyterian Medical Center has been limited by the lack of a flexible programming environment that supports multiple client user platforms. The World Wide Web offers a potential solution, with its multifunction servers, multiplatform clients, and use of standard protocols for displaying information. The authors are now using the Web, coupled with their own local clinical data server and vocabulary server, to carry out rapid prototype development of clinical information systems. They have developed one such prototype system that can be run on most popular computing platforms from anywhere on the Internet. The Web paradigm allows easy integration of clinical information with other local and Internet-based information sources. The Web also simplifies many aspects of application design; for example, it includes facilities for the use of encryption to meet the authors' security and confidentiality requirements. The prototype currently runs on only the Web server in the Department of Medical Informatics at Columbia University, but it could be run on other Web servers that access the authors' clinical data and vocabulary servers. It could also be adapted to access clinical information from other systems with similar server capabilities. This approach may be adaptable for use in developing institution-independent standards for data and application sharing.

OBJECTIVE: To develop a representational schema for clinical data for use in exchanging data and applications, using a collaborative approach. DESIGN: Representational models for clinical radiology were independently developed manually by several Canon Group members who had diverse application interests, using sample reports. These models were merged into one common model through an iterative process by means of workshops, meetings, and electronic mail. RESULTS: A core merged model for radiologic findings present in a set of reports that subsumed the models that were developed independently. CONCLUSIONS: The Canon Group's modeling effort focused on a collaborative approach to developing a representational schema for clinical concepts, using chest radiography reports as the initial experiment. This effort resulted in a core model that represents a consensus. Further efforts in modeling will extend the representational coverage and will also address issues such as scalability, automation, evaluation, and support of the collaborative effort.

Controlled medical vocabularies are at the heart of almost all health care computing applications. This article reviews the vocabularies available today and some of the reasons why they have failed to meet the needs of application developers. Also described is some of the current work being done to address shortcomings in previous vocabulary content and structure. © 1995 John Wiley & Sons, Inc.

James J. Cimino, MD; Controlled Medical Vocabulary Construction: Methods from the Canon Group, Journal of the American Medical Informatics Association, Volume 1

OBJECTIVE: Develop a knowledge-based representation for a controlled terminology of clinical information to facilitate creation, maintenance, and use of the terminology. DESIGN: The Medical Entities Dictionary (MED) is a semantic network, based on the Unified Medical Language System (UMLS), with a directed acyclic graph to represent multiple hierarchies. Terms from four hospital systems (laboratory, electrocardiography, medical records coding, and pharmacy) were added as nodes in the network. Additional knowledge about terms, added as semantic links, was used to assist in integration, harmonization, and automated classification of disparate terminologies. RESULTS: The MED contains 32,767 terms and is in active clinical use. Automated classification was successfully applied to terms for laboratory specimens, laboratory tests, and medications. One benefit of the approach has been the automated inclusion of medications into multiple pharmacologic and allergenic classes that were not present in the pharmacy system. Another benefit has been the reduction of maintenance efforts by 90%. CONCLUSION: The MED is a hybrid of terminology and knowledge. It provides domain coverage, synonymy, consistency of views, explicit relationships, and multiple classification while preventing redundancy, ambiguity (homonymy) and misclassification.

The Canon Group is an informal organization of medical informatics researchers who are working on the problem of developing a "deeper" representation formalism for use in exchanging data and developing applications. Individuals in the group represent experts in such areas as knowledge representation and computational linguistics, as well as in a variety of medical subdisciplines. All share the view that current mechanisms for the characterization of medical phenomena are either inadequate (limited or rigid) or idiosyncratic (useful for a specific application but incapable of being generalized or extended). The Group proposes to focus on the design of a general schema for medical-language representation including the specification of the resources and associated procedures required to map language (including standard terminologies) into representations that make all implicit relations "visible," reveal "hidden attributes," and generally resolve ambiguous or vague references. The Group is proceeding by examining large numbers of texts (records) in medical sub-domains to identify candidate "concepts" and by attempting to develop general rules and representations for elements such as attributes and values so that all concepts may be expressed uniformly.

OBJECTIVE: Development of a general natural-language processor that identifies clinical information in narrative reports and maps that information into a structured representation containing clinical terms. DESIGN: The natural-language processor provides three phases of processing, all of which are driven by different knowledge sources. The first phase performs the parsing. It identifies the structure of the text through use of a grammar that defines semantic patterns and a target form. The second phase, regularization, standardizes the terms in the initial target structure via a compositional mapping of multi-word phrases. The third phase, encoding, maps the terms to a controlled vocabulary. Radiology is the test domain for the processor and the target structure is a formal model for representing clinical information in that domain. MEASUREMENTS: The impression sections of 230 radiology reports were encoded by the processor. Results of an automated query of the resultant database for the occurrences of four diseases were compared with the analysis of a panel of three physicians to determine recall and precision. RESULTS: Without training specific to the four diseases, recall and precision of the system (combined effect of the processor and query generator) were 70% and 87%. Training of the query component increased recall to 85% without changing precision.

OBJECTIVE: Develop a representational schema for clinical concepts and apply it to the task of encoding radiology reports of the chest. DESIGN: The schema was developed following a manual analysis of sample reports from the domain. The schema has two main components: the Medical Entities Dictionary (MED), which specifies the formal representation of the concepts in the domain and of their structures, and the natural-language processor, which specifies the linguistic expressions of the concepts. The schema was evaluated by applying it to a test set of 7,500 reports. Two-hundred reports from the test set were manually analyzed by a medical expert to determine the accuracy and success rate of the system. RESULTS: 82% of the 7,500 reports that contained relevant clinical information were successfully structured automatically. For the smaller set of 200 reports, 80% were structured successfully with an accuracy rate of 97%. CONCLUSIONS: The schema is a formal representation for clinical concepts in radiology reports, and provides domain coverage that is particularly well-suited for natural-language processing of radiology for use in a decision support system.