The AMIA biomedical informatics (BMI) core competencies have been designed to support and guide graduate education in BMI, the core scientific discipline underlying the breadth of the field's research, practice, and education. The core definition of BMI adopted by AMIA specifies that BMI is 'the interdisciplinary field that studies and pursues the effective uses of biomedical data, information, and knowledge for scientific inquiry, problem solving and decision making, motivated by efforts to improve human health.' Application areas range from bioinformatics to clinical and public health informatics and span the spectrum from the molecular to population levels of health and biomedicine. The shared core informatics competencies of BMI draw on the practical experience of many specific informatics sub-disciplines. The AMIA BMI analysis highlights the central shared set of competencies that should guide curriculum design and that graduate students should be expected to master.

OBJECTIVE: Standardized surveillance syndromes do not exist but would facilitate sharing data among surveillance systems and comparing the accuracy of existing systems. The objective of this study was to create reference syndrome definitions from a consensus of investigators who currently have or are building syndromic surveillance systems. DESIGN: Clinical condition-syndrome pairs were catalogued for 10 surveillance systems across the United States and the representatives of these systems were brought together for a workshop to discuss consensus syndrome definitions. RESULTS: Consensus syndrome definitions were generated for the four syndromes monitored by the majority of the 10 participating surveillance systems: Respiratory, gastrointestinal, constitutional, and influenza-like illness (ILI). An important element in coming to consensus quickly was the development of a sensitive and specific definition for respiratory and gastrointestinal syndromes. After the workshop, the definitions were refined and supplemented with keywords and regular expressions, the keywords were mapped to standard vocabularies, and a web ontology language (OWL) ontology was created. LIMITATIONS: The consensus definitions have not yet been validated through implementation. CONCLUSION: The consensus definitions provide an explicit description of the current state-of-the-art syndromes used in automated surveillance, which can subsequently be systematically evaluated against real data to improve the definitions. The method for creating consensus definitions could be applied to other domains that have diverse existing definitions.

BioProspecting is a novel approach that enabled our team to mine data related to genetic markers from the New England Journal of Medicine (NEJM) utilizing SNOMED CT and the Human Gene Onotology (HUGO). The Biomedical Informatics Research Collaborative was able to link genes and disorders using the Multi-threaded Clinical Vocabulary Server (MCVS) and natural language processing engine, whose output creates an ontology-network using the semantic encodings of the literature that is organized by these two terminologies. We identified relationships between (genes or proteins) and (diseases or drugs) as linked by metabolic functions and identified potentially novel functional relationships between, for example, genes and diseases (e.g. Article #1 ([Gene - IL27] = > {Enzyme - Dipeptidyl Carboxypeptidase 1}) and Article #2 ({Enzyme - Dipeptidyl Carboxypeptidase 1} < = [Disorder - Type II DM]) showing a metabolic link between IL27 and Type II DM). In this manuscript we describe our method for developing the database and its content as well as its potential to assist in the discovery of novel markers and drugs.

OBJECTIVE: Interface terminologies are designed to support interactions between humans and structured medical information. In particular, many interface terminologies have been developed for structured computer based documentation systems. Experts and policy-makers have recommended that interface terminologies be mapped to reference terminologies. The goal of the current study was to evaluate how well the reference terminology SNOMED CT could map to and represent two interface terminologies, MEDCIN and the Categorical Health Information Structured Lexicon (CHISL). DESIGN: Automated mappings between SNOMED CT and 500 terms from each of the two interface terminologies were evaluated by human reviewers, who also searched SNOMED CT to identify better mappings when this was judged to be necessary. Reviewers judged whether they believed the interface terms to be clinically appropriate, whether the terms were covered by SNOMED CT concepts and whether the terms' implied semantic structure could be represented by SNOMED CT. MEASUREMENTS: Outcomes included concept coverage by SNOMED CT for study terms and their implied semantics. Agreement statistics and compositionality measures were calculated. RESULTS: The SNOMED CT terminology contained concepts to represent 92.4% of MEDCIN and 95.9% of CHISL terms. Semantic structures implied by study terms were less well covered, with some complex compositional expressions requiring semantics not present in SNOMED CT. Among sampled terms, those from MEDCIN were more complex than those from CHISL, containing an average 3.8 versus 1.8 atomic concepts respectively, p<0.001. CONCLUSION: Our findings support using SNOMED CT to provide standardized representations of information created using these two terminologies, but suggest that enriching SNOMED CT semantics would improve representation of the external terms.

OBJECTIVE: Evaluations of individual terminology systems should be driven in part by the intended usages of such systems. Clinical interface terminologies support interactions between healthcare providers and computer-based applications. They aid practitioners in converting clinical "free text" thoughts into the structured, formal data representations used internally by application programs. Interface terminologies also serve the important role of presenting existing stored, encoded data to end users in human-understandable and actionable formats. The authors present a model for evaluating functional utility of interface terminologies based on these intended uses. DESIGN: Specific parameters defined in the manuscript comprise the metrics for the evaluation model. MEASUREMENTS: Parameters include concept accuracy, term expressivity, degree of semantic consistency for term construction and selection, adequacy of assertional knowledge supporting concepts, degree of complexity of pre-coordinated concepts, and the "human readability" of the terminology. The fundamental metric is how well the interface terminology performs in supporting correct, complete, and efficient data encoding or review by humans. RESULTS: Authors provide examples demonstrating performance of the proposed evaluation model in selected instances. CONCLUSION: A formal evaluation model will permit investigators to evaluate interface terminologies using a consistent and principled approach. Terminology developers and evaluators can apply the proposed model to identify areas for improving interface terminologies.

Previous investigators have defined clinical interface terminology as a systematic collection of health care-related phrases (terms) that supports clinicians' entry of patient-related information into computer programs, such as clinical "note capture" and decision support tools. Interface terminologies also can facilitate display of computer-stored patient information to clinician-users. Interface terminologies "interface" between clinicians' own unfettered, colloquial conceptualizations of patient descriptors and the more structured, coded internal data elements used by specific health care application programs. The intended uses of a terminology determine its conceptual underpinnings, structure, and content. As a result, the desiderata for interface terminologies differ from desiderata for health care-related terminologies used for storage (e.g., SNOMED-CT), information retrieval (e.g., MeSH), and classification (e.g., ICD9-CM). Necessary but not sufficient attributes for an interface terminology include adequate synonym coverage, presence of relevant assertional knowledge, and a balance between pre- and post-coordination. To place interface terminologies in context, this article reviews historical goals and challenges of clinical terminology development in general and then focuses on the unique features of interface terminologies.

Continuing education and improvement of medical practice has long been a tradition in the medical profession. As John Shaw Billings noted over a century ago, “The education of the doctor which goes on after he has his degree is, after all, the most important part of his education.” 1 Beyond the extensive array of formal continuing education programs available, physicians have, over the years, developed a variety of informal approaches to improving clinical skills based on review and critique of patient management. 2 A key ingredient of this tradition has been a focus on recognizing and learning from medical error. The 13th Century Oath of Maimonides advises physicians: “Today he can discover his errors of yesterday and tomorrow he can obtain a new light on what he thinks himself sure of today.” Continuing this tradition, Sir William Osler advised young physicians, “Begin early to make a three-fold category—clear cases, doubtful cases and mistakes…. It is only by getting your cases grouped in this way that you can make any real progress in your post-collegiate education.” 3 Osler himself served as a role model in this regard, as Cushing describes: Once in a ward class there was a man whom he demonstrated as showing all the classical symptoms of croupous pneumonia. The man came to autopsy later. He had no pneumonia, but a chest full of fluid. Dr. Osler seemed delighted, sent especially for all those in his ward class, showed them what a mistake he had made, how it might have been avoided and how careful they should be not to repeat it. 4 Recently the profession's interest in recognizing, learning from, and preventing medical error has been reinvigorated, as the papers in this special issue of JAMIA illustrate. In 1998, the President's Advisory Commission on Consumer Protection and Quality in the Health Care Industry presented its final report on quality to the President of the United States. Reduction of medical errors was listed as one of it's top priorities. 5 Although a nascent science in which much remains to be learned, the applications of human factors engineering, systems science, information science, and computer technology are widely acknowledged to hold great promise for reducing medical error and improving patient safety. These converging disciplines, combined with advances in our understanding of how continuing education can be designed to change clinical practice, 6 offer an opportunity to effectively refocus physicians' continuing education on medical error and patient safety. Two key features of modern medical care that must be recognized to understand and address medical error are that it is multidisciplinary and that it occurs within a complex, hierarchically organized system. 7 To be most effective, a program of continuing professional education on medical error must also reflect these features. At the VHA National Center for Patient Safety in Ann Arbor, Michigan, participants were faced with the goal of providing training and tools to over 172 healthcare facilities. 8 Their educational efforts enabled healthcare providers to use root cause analysis for adverse events and close calls and to better their statistics by identifying ADEs in 31% of the patients sampled. First, because care is necessarily multidisciplinary, continuing education of professionals independent of one another can have only limited effect. New models of interdisciplinary training, incorporating principles of crew resource management pioneered in aviation, have already been instituted and hold great promise. Second, because patients are cared for in complex, hierarchically organized systems, continuing education must be designed to address learning needs at multiple levels, from individuals to small teams and groups, interacting groups, the larger organization, and so on. To be helpful to patients, continuing professional education must affect not only the learners' knowledge and skills, but also their actual practices. Although conventional “sit and listen” continuing medical education (CME) has been disappointing in terms of its impact on clinical practice, 6 , 9 much has been learned in recent years about the elements of CME that do have an impact. 10 , 11 In particular, according to the review by Davis et al., CME programs that were effective tended to have these features: (1) focused on identified gaps in knowledge or practice (and need to change); (2) addressed barriers to change in the practice environment; (3) included practice enabling strategies; and (4) provided opportunities for rehearsal and reinforcement. 6 The recommendations contained in the IOM Report “To Err is Human” constructed a four-tiered approach: Establishing a national focus to create leadership, research, tools and protocols to enhance the knowledge base about safety; Identifying and learning from errors through the immediate and strong mandatory reporting efforts, as well as the encouragement of voluntary efforts, both with the aim of making sure the system continues to be made safer for patients; Raising standards and expectations for improvements in safety through the actions of oversight organizations, group purchasers, and professional groups; and Creating safety systems inside health care organizations through the implementation of safe practices at the delivery level. This level is the ultimate target of all the recommendations. 12 We suggest that a fifth recommendation be added, which is educating our future clinicians with regard to systematic error as well as how to avoid such error and the associated adverse events. A core patient safety curriculum for education of clinicians needs to be created. Table 1 offers an example of a potential patient safety curriculum. This table contains examples of topics that are likely to be covered in a patient safety CME program. Adverse drug events This module discusses the known mechanisms by which ADEs occur. Current and innovative methods for identifying sources of systematic errors need to be impressed on clinicians. The differences between error, adverse events, and harm must be taught. Formulas for best practice in medication prescribing (e.g., always check allergies, write legibly, check liver and renal functions when appropriate) need to be analyzed. Order entry for medications, bar coding of medications, and drug administration data collection are an important part of the curriculum. Errors of omission The method for showing clinicians how they can be certain not to forget what they are supposed to do in health care falls roughly into two categories. The first is knowing what is your personal responsibility (i.e., knowing your medicine and your job); the second is knowing how to organize your life so that you remember to do the things that you are supposed to accomplish. For the former, the curriculum should clearly lay out the responsibility of every member of the team in the patient care process. This must serve as a guide for individual members of the team to know their responsibilities and the responsibilities of their coworkers. The second objective provides guidance in documenting a clinician's to-do list, effectively signing out patients to a coworker when appropriate, transmitting the needed patient information effectively whenever required, and related activities. Errors of commission How sure is sure? When do you know enough to take responsibility for a decision in medicine? How do you get back-up when you are unsure? How do you recognize that an error has been made? What is your responsibility with regard to correcting an error, if possible? What documentation is required to comply with patient incident and sentinel event reporting? Our curriculum must address these and other critical issues in this arena. Discharge planning The discharge planning process is discussed in detail. Documenting drugs and their proper use is always of utmost importance. This responsibility is extremely critical when patients are transferring to nursing homes because the discharge summary is used as orders for medications. Selecting a safe environment for a patient leaving the hospital can be challenging, and helpful procedures are discussed during this module. 25 Discharge summaries vary in quality; therefore, essential features of a useful and usable discharge summary are identified. Transitions in level of care Going from the outpatient setting to the inpatient setting is often haphazard. Courses must be developed that teach the basic principles that should allow clinicians to make cogent judgments about the need for hospitalization. Similarly, transitioning to the outpatient setting in a safe and effective manner has certain associated principles that need to be employed to teach this module (e.g., safe mobility around the home for home-going patients to prevent falls in the elderly). Consultation Knowing when to consult a subspecialist for a particular medical condition is part of the art of medicine. However, some of the variation in practice is based on a clinician's predispositions rather than the patient's must a basic understanding of the principles that the most clinicians use in making these for this module which be best to a and the most to a that you are not to which after or during are one of often adverse events. must be the basic principles in a medical Safety of The safety of prescribing to patients take is of The should and for clinicians the important principles the of when patients are on or that as a do not This module must also to information at the of which clinicians to practice when prescribing medications. Adverse drug events This module discusses the known mechanisms by which ADEs occur. Current and innovative methods for identifying sources of systematic errors need to be impressed on clinicians. The differences between error, adverse events, and harm must be taught. Formulas for best practice in medication prescribing (e.g., always check allergies, write legibly, check liver and renal functions when appropriate) need to be analyzed. Order entry for medications, bar coding of medications, and drug administration data collection are an important part of the curriculum. Errors of omission The method for showing clinicians how they can be certain not to forget what they are supposed to do in health care falls roughly into two categories. The first is knowing what is your personal responsibility (i.e., knowing your medicine and your job); the second is knowing how to organize your life so that you remember to do the things that you are supposed to accomplish. For the former, the curriculum should clearly lay out the responsibility of every member of the team in the patient care process. This must serve as a guide for individual members of the team to know their responsibilities and the responsibilities of their coworkers. The second objective provides guidance in documenting a clinician's to-do list, effectively signing out patients to a coworker when appropriate, transmitting the needed patient information effectively whenever required, and related activities. Errors of commission How sure is sure? When do you know enough to take responsibility for a decision in medicine? How do you get back-up when you are unsure? How do you recognize that an error has been made? What is your responsibility with regard to correcting an error, if possible? What documentation is required to comply with patient incident and sentinel event reporting? Our curriculum must address these and other critical issues in this arena. Discharge planning The discharge planning process is discussed in detail. Documenting drugs and their proper use is always of utmost importance. This responsibility is extremely critical when patients are transferring to nursing homes because the discharge summary is used as orders for medications. Selecting a safe environment for a patient leaving the hospital can be challenging, and helpful procedures are discussed during this module. 25 Discharge summaries vary in quality; therefore, essential features of a useful and usable discharge summary are identified. Transitions in level of care Going from the outpatient setting to the inpatient setting is often haphazard. Courses must be developed that teach the basic principles that should allow clinicians to make cogent judgments about the need for hospitalization. Similarly, transitioning to the outpatient setting in a safe and effective manner has certain associated principles that need to be employed to teach this module (e.g., safe mobility around the home for home-going patients to prevent falls in the elderly). Consultation Knowing when to consult a subspecialist for a particular medical condition is part of the art of medicine. However, some of the variation in practice is based on a clinician's predispositions rather than the patient's must a basic understanding of the principles that the most clinicians use in making these for this module which be best to a and the most to a that you are not to which after or during are one of often adverse events. must be the basic principles in a medical Safety of The safety of prescribing to patients take is of The should and for clinicians the important principles the of when patients are on or that as a do not This module must also to information at the of which clinicians to practice when prescribing medications. Adverse drug events This module discusses the known mechanisms by which ADEs occur. Current and innovative methods for identifying sources of systematic errors need to be impressed on clinicians. The differences between error, adverse events, and harm must be taught. Formulas for best practice in medication prescribing (e.g., always check allergies, write legibly, check liver and renal functions when appropriate) need to be analyzed. Order entry for medications, bar coding of medications, and drug administration data collection are an important part of the curriculum. Errors of omission The method for showing clinicians how they can be certain not to forget what they are supposed to do in health care falls roughly into two categories. The first is knowing what is your personal responsibility (i.e., knowing your medicine and your job); the second is knowing how to organize your life so that you remember to do the things that you are supposed to accomplish. For the former, the curriculum should clearly lay out the responsibility of every member of the team in the patient care process. This must serve as a guide for individual members of the team to know their responsibilities and the responsibilities of their coworkers. The second objective provides guidance in documenting a clinician's to-do list, effectively signing out patients to a coworker when appropriate, transmitting the needed patient information effectively whenever required, and related activities. Errors of commission How sure is sure? When do you know enough to take responsibility for a decision in medicine? How do you get back-up when you are unsure? How do you recognize that an error has been made? What is your responsibility with regard to correcting an error, if possible? What documentation is required to comply with patient incident and sentinel event reporting? Our curriculum must address these and other critical issues in this arena. Discharge planning The discharge planning process is discussed in detail. Documenting drugs and their proper use is always of utmost importance. This responsibility is extremely critical when patients are transferring to nursing homes because the discharge summary is used as orders for medications. Selecting a safe environment for a patient leaving the hospital can be challenging, and helpful procedures are discussed during this module. 25 Discharge summaries vary in quality; therefore, essential features of a useful and usable discharge summary are identified. Transitions in level of care Going from the outpatient setting to the inpatient setting is often haphazard. Courses must be developed that teach the basic principles that should allow clinicians to make cogent judgments about the need for hospitalization. Similarly, transitioning to the outpatient setting in a safe and effective manner has certain associated principles that need to be employed to teach this module (e.g., safe mobility around the home for home-going patients to prevent falls in the elderly). Consultation Knowing when to consult a subspecialist for a particular medical condition is part of the art of medicine. However, some of the variation in practice is based on a clinician's predispositions rather than the patient's must a basic understanding of the principles that the most clinicians use in making these for this module which be best to a and the most to a that you are not to which after or during are one of often adverse events. must be the basic principles in a medical Safety of The safety of prescribing to patients take is of The should and for clinicians the important principles the of when patients are on or that as a do not This module must also to information at the of which clinicians to practice when prescribing medications. Adverse drug events This module discusses the known mechanisms by which ADEs occur. Current and innovative methods for identifying sources of systematic errors need to be impressed on clinicians. The differences between error, adverse events, and harm must be taught. Formulas for best practice in medication prescribing (e.g., always check allergies, write legibly, check liver and renal functions when appropriate) need to be analyzed. Order entry for medications, bar coding of medications, and drug administration data collection are an important part of the curriculum. Errors of omission The method for showing clinicians how they can be certain not to forget what they are supposed to do in health care falls roughly into two categories. The first is knowing what is your personal responsibility (i.e., knowing your medicine and your job); the second is knowing how to organize your life so that you remember to do the things that you are supposed to accomplish. For the former, the curriculum should clearly lay out the responsibility of every member of the team in the patient care process. This must serve as a guide for individual members of the team to know their responsibilities and the responsibilities of their coworkers. The second objective provides guidance in documenting a clinician's to-do list, effectively signing out patients to a coworker when appropriate, transmitting the needed patient information effectively whenever required, and related activities. Errors of commission How sure is sure? When do you know enough to take responsibility for a decision in medicine? How do you get back-up when you are unsure? How do you recognize that an error has been made? What is your responsibility with regard to correcting an error, if possible? What documentation is required to comply with patient incident and sentinel event reporting? Our curriculum must address these and other critical issues in this arena. Discharge planning The discharge planning process is discussed in detail. Documenting drugs and their proper use is always of utmost importance. This responsibility is extremely critical when patients are transferring to nursing homes because the discharge summary is used as orders for medications. Selecting a safe environment for a patient leaving the hospital can be challenging, and helpful procedures are discussed during this module. 25 Discharge summaries vary in quality; therefore, essential features of a useful and usable discharge summary are identified. Transitions in level of care Going from the outpatient setting to the inpatient setting is often haphazard. Courses must be developed that teach the basic principles that should allow clinicians to make cogent judgments about the need for hospitalization. Similarly, transitioning to the outpatient setting in a safe and effective manner has certain associated principles that need to be employed to teach this module (e.g., safe mobility around the home for home-going patients to prevent falls in the elderly). Consultation Knowing when to consult a subspecialist for a particular medical condition is part of the art of medicine. However, some of the variation in practice is based on a clinician's predispositions rather than the patient's must a basic understanding of the principles that the most clinicians use in making these for this module which be best to a and the most to a that you are not to which after or during are one of often adverse events. must be the basic principles in a medical Safety of The safety of prescribing to patients take is of The should and for clinicians the important principles the of when patients are on or that as a do not This module must also to information at the of which clinicians to practice when prescribing medications. should patient safety be first and in the curriculum of continuing medical The for patient as are not only for medical and but also for clinicians. training with new advances in tools and clinical can make a A at the first “To Err is a Health in identified the to in as a of medical making them the fifth to cause of in the United States. These errors in than and In medication errors cause another The to the health care system is also The IOM that medical errors the with about of those associated with from this it is that medical errors are a national health that has in and The healthcare system must address this in the manner that it such as and The is to medical errors and patient as demonstrated by the efforts of the Quality and recent to in to for these 5 For these it is and that patient safety must a part of continuing medical education for and every of the of clinical information systems on patient safety In this issue to the of information event and and preventing adverse events is essential for improving medical and in is an to have ADEs through effective, this is and et the of adverse events within the of and how to use process to adverse events. These are one method for setting for a clinical such as during the of process should be part of a CME program. systems much of the for the decision needed to patient safety. and an to error. the use of to an between the patient's condition and a of adverse events, which are to the to serve as an As adverse event a part of clinical practice, clinicians need to be in the use and of et the to use a clinical to the of adverse events and their associated adverse event of in this is an for most of as clinicians. et for all based decision systems to take into the patient safety issues associated with their clinical For the recommendation for the use of drugs that can to renal should a to clinicians about this to the human factors issues patient safety systems, their impact be are to human need to understand that and take of the needs of our these systems et the needed between and This as a to the implementation of patient safety systems in the through a understanding of the human factors of nursing can a implementation in this environment occur. and the need to understand the environment and the of the and the on any patient safety system for this The of should be a part of any patient safety CME curriculum. et suggest the use of to address some of between and physicians in the inpatient As a curriculum in patient safety it is that need not only address the of systems that have been demonstrated to adverse events but also to clinicians but important that to adverse event such as between physicians and The of about adverse events in improving patient is et how the terms as the of an adverse in the are by organizations that care about patient safety. of the in the are no in their of these than Health and the and It that this issue needs to be effective knowledge management can (i.e., the of a for events and et the model of human to suggest to the to its to core patient safety in the This is a in the of and data for patient safety. et an innovative the Quality to in information about the of and of the of health is an important part of patient decision This the on et which is a and for management of The the issue of both the and the clinician's in patient is essential for patient that data can to patient patients not to in research, and not the root cause for patient safety. The by et that it is to data that have been with the which use the the potential to in et take this one by showing that an system that can (and them by is of in data identifying need to be developed that can and have enough to allow to to safety. of these important to health that are important for in a patient safety curriculum for continuing medical Our curriculum topics within the of the of patient safety as it the patient and the issues into that We that this method of is to on the of patient care This focus provides the of examples of the of health to patient safety

Usability evaluations are a powerful tool that can assist developers in their efforts to optimize the quality of their web environment. This underutilized, experimental method can serve to move applications toward true user-centered design. This article describes the usability methodology and illustrates its importance and application by describing a usability study undertaken at the Mayo Clinic for the purpose of improving an academic research web environment. Academic institutions struggling in an era of declining reimbursements are finding it difficult to maintain academic enterprises on the back of clinical revenues. This may result in declining amounts of time that clinical investigators have to spend in non-patient-related activities. For this reason, we have undertaken to design a web environment, which can minimize the time that a clinician-investigator needs to spend to accomplish academic instrumental activities of daily living. Usability evaluation is a powerful application of human factors engineering, which can improve the utility of web-based Informatics applications.

OBJECTIVE: The Computerized Patient Record System is deployed at all 173 Veterans Affairs (VA) medical centers. Providers access clinical notes in the system from a note title menu. Following its implementation at the Nashville VA Medical Center, users expressed dissatisfaction with the time required find notes among hundreds of irregularly structured titles. The authors' objective was to develop a document-naming nomenclature (DNN) that creates informative, structured note titles that improve information access. DESIGN: One thousand ninety-four unique note titles from two VA medical centers were reviewed. A note-naming nomenclature and compositional syntax were derived. Compositional order was determined by user preference survey. MEASUREMENTS: The DNN was evaluated by modeling note titles from the Salt Lake City VA Medical Center (n=877), Vanderbilt University Medical Center (n=554), and the Mayo Clinic (n=42). A preliminary usability evaluation was conducted on a structured title display and sorting application. RESULTS: Classes of note title components were found by inspection. Components describe characteristics of the author, the health care event, and the organizational unit providing care. Terms were taken from VA medical center information systems and national standards. The DNN model accurately described 97 to 99 percent of note titles from the test sites. The DNN term coverage varied, depending on component and site. Users found the DNN title format useful and the DNN-based title sorting and note review application easy to learn and quick to use. CONCLUSION: The DNN accurately models note titles at five medical centers. Preliminary usability data indicate that DNN integration with title parsing and sorting software enhances information access.

To the Editor:—In the recently published study by Nadkarni et al.,1 the authors used text-mining software to extract concepts from clinical documents. Matching of these concepts was attempted with the UMLS 99 Metathesaurus. Matches were then categorized as true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN) from 8,745 terms in a “training set” and 1,701 terms in a “test set,” for a total of 10,446 terms. True positives were reported as 82.6 percent for the training set and 76.3 percent for the test set. In 1999, we carried out an almost identical study using the identical version of the UMLS, on a larger scale, which resulted in very similar results that were presented at the 1999 AMIA Annual Symposium.2 In our study, 4,994 of the most frequently referenced terms were chosen from 1,000,000 terms randomly extracted from the general Mayo Clinic Master Sheet Index and the Impression/Report/Plan section of the Mayo Clinic clinical notes system, to form a general medicine set. The Mayo Clinic Department of Dermatology independently developed a lexicon of 9,050 unique terms describing lesions photographed in their practice, which formed a specialty-specific set. We used automated term composition and the UMLS to assess match rates. In addition, we looked at match rates on our total 14,044 terms based on filtering using the UMLS semantic types. Comparison of the data from the two studies (Table1) reveals striking similarities. Comparison of Data Comparison of Data What we recognized in 1999, which was omitted from the analysis of Nadkarni et al., was that other metrics are important in the clinical interpretation of these data. Representing the data as shown in Table1 allows for useful combinations. The true-positive rate is the number of true positives divided by the sum of true positives and false negatives (TP/[TP+FN]), yielding sensitivity. Similar calculation of specificity (TN/[TN+FP]), positive predictive value (TP/[TP+FP]), and positive likelihood ratio (sensitivity/[1−specificity]) can be carried out. When these combinations are done, it is evident that concept matching in the UMLS is actually much better than was implied by the only true-positive incidence quoted by Nadkarni et al. (Table 2). Comparison of Metrics Comparison of Metrics The differences in these metrics across the data sets are related to the relatively liberal definition of true positives and the relatively strict definition of false positives given by Nadkarni et al. Unlike them, we did take negation into account when determining true positives. Their definition of false positive was limited to acronyms, abbreviations, spelling/grammar errors, and proper names, whereas we had each “match” judged by a practicing internist to make the determination of true positive or false positive, regardless of term classification. By applying automated term composition with filters based on the UMLS semantic types, we showed that we could balance sensitivity and specificity to optimize the other metrics (Table 3). Metrics Derived by Use of Semantic Type Filtering Metrics Derived by Use of Semantic Type Filtering Our study publication predates that of Nadkarni et al. by almost 14 months. It is clear from the data of both studies that concept indexing with the UMLS is actually highly sensitive, with quite a high positive predictive value. This conclusion was omitted by Nadkarni et al. but is worthy of further analysis as the UMLS, and the algorithms that use it, increase their specificity to match the already quite excellent sensitivity. Surely, for a vocabulary to be useful it must evolve and its content must grow. The UMLS now contains more than 700,000 concepts, but it still does not cover all clinically useful terminology. As Cimino3 states, “…a formal methodology is needed for expanding content.” Chute et al.4 reinforce this statement with the argument that “in the absence of a single, all-embracing health care terminology, there need to be coordination and organizing support for interrelated terminologies…” and that “developers of clinical classifications must consider ways they can develop their systems to become part of an integrated set of terminology systems.” If terms are added to a vocabulary indiscriminately, however, redundancy and combinatorial explosion may make the vocabulary unwieldy and difficult to search in a timely fashion. “An alternative approach is to enumerate all the atoms of a terminology and allow users to combine them into necessary coded terms, allowing compositional extensibility.”3,5,6 One risk of this approach is its potential for making the use of the vocabulary more complex. We hypothesized that automated term composition as developed and tested in a randomized controlled trial7 would allow large-scale coverage of specialty-specific and general local vocabularies. This automated process would facilitate the appropriate inclusion of such terms into a larger vocabulary without creating redundancy. As we noted in 1999, user-directed composition may allow salvage of many of the false-positive and true-negative matches, thus significantly increasing the incorporation rate.8 The true-negative terms, which do not yield to user-directed post-coordination of concepts to form a positive exact match, could form a set of terms that could be considered for incorporation into larger vocabularies without the onus of redundancy. Given the large size of both the specialty-specific and local general terminological corpi used in our study, this method should be generalizable to other local specialty-specific and general terminology sets. These results help solidify the need for compositional mechanisms for terminological representation and show the utility of the considerable synonymy offered by the UMLS. Future research should focus on how to integrate colloquial terminologies such as the UMLS with formal reference terminologies.

Nursing Vocabulary Summit participants were challenged to consider whether reference terminology and information models might be a way to move toward better capture of data in electronic medical records. A requirement of such reference models is fidelity to representations of domain knowledge. This article discusses embedded structures in three different approaches to organizing domain knowledge: scientific reasoning, expertise, and standardized nursing languages. The concept of pressure ulcer is presented as an example of the various ways lexical elements used in relation to a specific concept are organized across systems. Different approaches to structuring information-the clinical information system, minimum data sets, and standardized messaging formats-are similarly discussed. Recommendations include identification of the polyhierarchies and categorical structures required within a reference terminology, systematic evaluations of the extent to which structured information accurately and completely represents domain knowledge, and modifications or extensions to existing multidisciplinary efforts.