OBJECTIVE: Use heuristic, deep learning (DL), and hybrid AI methods to predict semantic group (SG) assignments for new UMLS Metathesaurus atoms, with target accuracy ≥95%. MATERIALS AND METHODS: We used train-test datasets from successive 2020AA-2022AB UMLS Metathesaurus releases. Our heuristic "waterfall" approach employed a sequence of 7 different SG prediction methods. Atoms not qualifying for a method were passed on to the next method. The DL approach generated BioWordVec and SapBERT embeddings for atom names, BioWordVec embeddings for source vocabulary names, and BioWordVec embeddings for atom names of the second-to-top nodes of an atom's source hierarchy. We fed a concatenation of the 4 embeddings into a fully connected multilayer neural network with an output layer of 15 nodes (one for each SG). For both approaches, we developed methods to estimate the probability that their predicted SG for an atom would be correct. Based on these estimations, we developed 2 hybrid SG prediction methods combining the strengths of heuristic and DL methods. RESULTS: The heuristic waterfall approach accurately predicted 94.3% of SGs for 1 563 692 new unseen atoms. The DL accuracy on the same dataset was also 94.3%. The hybrid approaches achieved an average accuracy of 96.5%. CONCLUSION: Our study demonstrated that AI methods can predict SG assignments for new UMLS atoms with sufficient accuracy to be potentially useful as an intermediate step in the time-consuming task of assigning new atoms to UMLS concepts. We showed that for SG prediction, combining heuristic methods and DL methods can produce better results than either alone.

In this editorial, the first 4 Editors-in-Chief of the Journal of the American Medical Informatics Association (JAMIA) reflect on its history and future.The origins and characterization of each Editor's era represent the "lived experience" of each Editor rather than a comparison of common metrics over time.We also qualitatively assess JAMIA's progress in meeting its original vision and goals, and posit considerations for its future.Table 1 summarizes key JAMIA-related events.

A panel sponsored by the American College of Medical Informatics (ACMI) at the 2021 AMIA Symposium addressed the provocative question: "Are Electronic Health Records dumbing down clinicians?" After reviewing electronic health record (EHR) development and evolution, the panel discussed how EHR use can impair care delivery. Both suboptimal functionality during EHR use and longer-term effects outside of EHR use can reduce clinicians' efficiencies, reasoning abilities, and knowledge. Panel members explored potential solutions to problems discussed. Progress will require significant engagement from clinician-users, educators, health systems, commercial vendors, regulators, and policy makers. Future EHR systems must become more user-focused and scalable and enable providers to work smarter to deliver improved care.

Over a 31-year span as Director of the US National Library of Medicine (NLM), Donald A.B. Lindberg, MD, and his extraordinary NLM colleagues fundamentally changed the field of biomedical and health informatics-with a resulting impact on biomedicine that is much broader than its influence on any single subfield. This article provides substance to bolster that claim. The review is based in part on the informatics section of a new book, "Transforming biomedical informatics and health information access: Don Lindberg and the US National Library of Medicine" (IOS Press, forthcoming 2021). After providing insights into selected aspects of the book's informatics-related contents, the authors discuss the broader context in which Dr. Lindberg and the NLM accomplished their transformative work.

OBJECTIVE: The goal of this study was to develop a practical framework for recognizing and disambiguating clinical abbreviations, thereby improving current clinical natural language processing (NLP) systems' capability to handle abbreviations in clinical narratives. METHODS: We developed an open-source framework for clinical abbreviation recognition and disambiguation (CARD) that leverages our previously developed methods, including: (1) machine learning based approaches to recognize abbreviations from a clinical corpus, (2) clustering-based semiautomated methods to generate possible senses of abbreviations, and (3) profile-based word sense disambiguation methods for clinical abbreviations. We applied CARD to clinical corpora from Vanderbilt University Medical Center (VUMC) and generated 2 comprehensive sense inventories for abbreviations in discharge summaries and clinic visit notes. Furthermore, we developed a wrapper that integrates CARD with MetaMap, a widely used general clinical NLP system. RESULTS AND CONCLUSION: CARD detected 27 317 and 107 303 distinct abbreviations from discharge summaries and clinic visit notes, respectively. Two sense inventories were constructed for the 1000 most frequent abbreviations in these 2 corpora. Using the sense inventories created from discharge summaries, CARD achieved an F1 score of 0.755 for identifying and disambiguating all abbreviations in a corpus from the VUMC discharge summaries, which is superior to MetaMap and Apache's clinical Text Analysis Knowledge Extraction System (cTAKES). Using additional external corpora, we also demonstrated that the MetaMap-CARD wrapper improved MetaMap's performance in recognizing disorder entities in clinical notes. The CARD framework, 2 sense inventories, and the wrapper for MetaMap are publicly available at https://sbmi.uth.edu/ccb/resources/abbreviation.htm . We believe the CARD framework can be a valuable resource for improving abbreviation identification in clinical NLP systems.

Real-time alerting systems typically warn providers about abnormal laboratory results or medication interactions. For more complex tasks, institutions create site-wide 'data warehouses' to support quality audits and longitudinal research. Sophisticated systems like i2b2 or Stanford's STRIDE utilize data warehouses to identify cohorts for research and quality monitoring. However, substantial resources are required to install and maintain such systems. For more modest goals, an organization desiring merely to identify patients with 'isolation' orders, or to determine patients' eligibility for clinical trials, may adopt a simpler, limited approach based on processing the output of one clinical system, and not a data warehouse. We describe a limited, order-entry-based, real-time 'pick off' tool, utilizing public domain software (PHP, MySQL). Through a web interface the tool assists users in constructing complex order-related queries and auto-generates corresponding database queries that can be executed at recurring intervals. We describe successful application of the tool for research and quality monitoring.

OBJECTIVE: To try to lower patient re-identification risks for biomedical research databases containing laboratory test results while also minimizing changes in clinical data interpretation. MATERIALS AND METHODS: In our threat model, an attacker obtains 5-7 laboratory results from one patient and uses them as a search key to discover the corresponding record in a de-identified biomedical research database. To test our models, the existing Vanderbilt TIME database of 8.5 million Safe Harbor de-identified laboratory results from 61 280 patients was used. The uniqueness of unaltered laboratory results in the dataset was examined, and then two data perturbation models were applied-simple random offsets and an expert-derived clinical meaning-preserving model. A rank-based re-identification algorithm to mimic an attack was used. The re-identification risk and the retention of clinical meaning for each model's perturbed laboratory results were assessed. RESULTS: Differences in re-identification rates between the algorithms were small despite substantial divergence in altered clinical meaning. The expert algorithm maintained the clinical meaning of laboratory results better (affecting up to 4% of test results) than simple perturbation (affecting up to 26%). DISCUSSION AND CONCLUSION: With growing impetus for sharing clinical data for research, and in view of healthcare-related federal privacy regulation, methods to mitigate risks of re-identification are important. A practical, expert-derived perturbation algorithm that demonstrated potential utility was developed. Similar approaches might enable administrators to select data protection scheme parameters that meet their preferences in the trade-off between the protection of privacy and the retention of clinical meaning of shared data.

OBJECTIVE: Medication safety requires that each drug be monitored throughout its market life as early detection of adverse drug reactions (ADRs) can lead to alerts that prevent patient harm. Recently, electronic medical records (EMRs) have emerged as a valuable resource for pharmacovigilance. This study examines the use of retrospective medication orders and inpatient laboratory results documented in the EMR to identify ADRs. METHODS: Using 12 years of EMR data from Vanderbilt University Medical Center (VUMC), we designed a study to correlate abnormal laboratory results with specific drug administrations by comparing the outcomes of a drug-exposed group and a matched unexposed group. We assessed the relative merits of six pharmacovigilance measures used in spontaneous reporting systems (SRSs): proportional reporting ratio (PRR), reporting OR (ROR), Yule's Q (YULE), the χ(2) test (CHI), Bayesian confidence propagation neural networks (BCPNN), and a gamma Poisson shrinker (GPS). RESULTS: We systematically evaluated the methods on two independently constructed reference standard datasets of drug-event pairs. The dataset of Yoon et al contained 470 drug-event pairs (10 drugs and 47 laboratory abnormalities). Using VUMC's EMR, we created another dataset of 378 drug-event pairs (nine drugs and 42 laboratory abnormalities). Evaluation on our reference standard showed that CHI, ROR, PRR, and YULE all had the same F score (62%). When the reference standard of Yoon et al was used, ROR had the best F score of 68%, with 77% precision and 61% recall. CONCLUSIONS: Results suggest that EMR-derived laboratory measurements and medication orders can help to validate previously reported ADRs, and detect new ADRs.

OBJECTIVE: Alerting systems, a type of clinical decision support, are increasingly prevalent in healthcare, yet few studies have concurrently measured the appropriateness of alerts with provider responses to alerts. Recent reports of suboptimal alert system design and implementation highlight the need for better evaluation to inform future designs. The authors present a comprehensive framework for evaluating the clinical appropriateness of synchronous, interruptive medication safety alerts. METHODS: Through literature review and iterative testing, metrics were developed that describe successes, justifiable overrides, provider non-adherence, and unintended adverse consequences of clinical decision support alerts. The framework was validated by applying it to a medication alerting system for patients with acute kidney injury (AKI). RESULTS: Through expert review, the framework assesses each alert episode for appropriateness of the alert display and the necessity and urgency of a clinical response. Primary outcomes of the framework include the false positive alert rate, alert override rate, provider non-adherence rate, and rate of provider response appropriateness. Application of the framework to evaluate an existing AKI medication alerting system provided a more complete understanding of the process outcomes measured in the AKI medication alerting system. The authors confirmed that previous alerts and provider responses were most often appropriate. CONCLUSION: The new evaluation model offers a potentially effective method for assessing the clinical appropriateness of synchronous interruptive medication alerts prior to evaluating patient outcomes in a comparative trial. More work can determine the generalizability of the framework for use in other settings and other alert types.

OBJECTIVE: The authors evaluated algorithms commonly used in syndromic surveillance for use as screening tools to detect potentially clonal outbreaks for review by infection control practitioners. DESIGN: Study phase 1 applied four aberrancy detection algorithms (CUSUM, EWMA, space-time scan statistic, and WSARE) to retrospective microbiologic culture data, producing a list of past candidate outbreak clusters. In phase 2, four infectious disease physicians categorized the phase 1 algorithm-identified clusters to ascertain algorithm performance. In phase 3, project members combined the algorithms to create a unified screening system and conducted a retrospective pilot evaluation. MEASUREMENTS: The study calculated recall and precision for each algorithm, and created precision-recall curves for various methods of combining the algorithms into a unified screening tool. RESULTS: Individual algorithm recall and precision ranged from 0.21 to 0.31 and from 0.053 to 0.29, respectively. Few candidate outbreak clusters were identified by more than one algorithm. The best method of combining the algorithms yielded an area under the precision-recall curve of 0.553. The phase 3 combined system detected all infection control-confirmed outbreaks during the retrospective evaluation period. LIMITATIONS: Lack of phase 2 reviewers' agreement indicates that subjective expert review was an imperfect gold standard. Less conservative filtering of culture results and alternate parameter selection for each algorithm might have improved algorithm performance. CONCLUSION: Hospital outbreak detection presents different challenges than traditional syndromic surveillance. Nevertheless, algorithms developed for syndromic surveillance have potential to form the basis of a combined system that might perform clinically useful hospital outbreak screening.

Colorectal cancer (CRC) screening rates are low despite confirmed benefits. The authors investigated the use of natural language processing (NLP) to identify previous colonoscopy screening in electronic records from a random sample of 200 patients at least 50 years old. The authors developed algorithms to recognize temporal expressions and 'status indicators', such as 'patient refused', or 'test scheduled'. The new methods were added to the existing KnowledgeMap concept identifier system, and the resulting system was used to parse electronic medical records (EMR) to detect completed colonoscopies. Using as the 'gold standard' expert physicians' manual review of EMR notes, the system identified timing references with a recall of 0.91 and precision of 0.95, colonoscopy status indicators with a recall of 0.82 and precision of 0.95, and references to actually completed colonoscopies with recall of 0.93 and precision of 0.95. The system was superior to using colonoscopy billing codes alone. Health services researchers and clinicians may find NLP a useful adjunct to traditional methods to detect CRC screening status. Further investigations must validate extension of NLP approaches for other types of CRC screening applications.

Dr Randolph A Miller begins a self-imposed JAMIA retirement on January 1, 2011 after serving as Editor-in-Chief for eight and a half years. He lauds the selection of Lucia Ohno-Machado as an energetic, innovative, and highly qualified successor.

Rapid technological change has affected many aspects of our society, but perhaps none more profoundly than the world of publishing. Scientific journals largely have moved to primary publication online. Clinicians and researchers first learn of, and then read articles on the Internet—and print them locally when required. Thus, as all publishers, whether commercial or non-profit, have attempted to address online communities, they have had to reconsider their business models and track new opportunities. The end of the current five-year Journal of the American Medical Informatics Association (JAMIA) publisher's contract in 2009 afforded the American Medical Informatics Association (AMIA) the opportunity to reconsider seriously its plans for the future of the journal. Thus, in May 2008, following an invited presentation by the Editor-in-Chief regarding the current status, perceived opportunities, and important decisions to be made for JAMIA, the AMIA Board of Directors began work on the renewal of the publishing contract for its flagship journal. The AMIA Board charged a task force comprising representatives from the Board of Directors, the AMIA Publications Committee, AMIA Staff, the JAMIA Editor, and a seasoned scholarly publishing consultant to assess future publishing options and to then issue a request for proposals. The task force convened frequently (electronically) during the first few months of its work, to examine the overarching principles of the process, to study the current publishing model, and to explore alternative approaches to publishing the association's scholarly journal. As their first order of business, the task force conducted a survey of all AMIA members to solicit comments on a variety of possible approaches. The survey asked members to indicate their personal preferences for five different publishing models, which included combinations of various options of print, online, and open-access versions of JAMIA. The survey results indicated that AMIA members' most favored (70%) overall model was the combined availability of print and online JAMIA versions. The ‘open-access only’ model was rated least desirable, due to the high per-article publication costs that all authors would have to pay. In the autumn of 2008, the task force developed a request for proposals (RFP), heavily influenced by an analysis of the data from the member survey. The RFP was released in December 2008 and sent to 11 publishers. Five bidders submitted proposals in early 2009. The task force analyzed the proposals, taking into account those solutions that would advance JAMIA's leadership position in the field of biomedical informatics, and those that would implement best practices and best evolving technologies. The task force ranked proposals based on criteria covering vision, innovation, quality, cost, marketing, and support. It considered the ability of the publisher to provide extraordinary service to authors and to the Editorial Office, to deliver an exceptional product, and to assure diversification of publication revenues. Three publishers were ultimately selected as finalists to make presentations to the task force in early May 2009 in Bethesda, MD. In a series of sessions held on the same day, the three finalists presented overviews of their proposals and entertained clarifying questions from Publisher Selection Task Force members. Immediately after the candidate publishers' presentations, the task force met to select which publisher it would recommend to the AMIA Board of Directors as the next publisher of JAMIA. In order to make the best selection, the task force focused on 14 evaluation criteria (as developed by Morna Conway, AMIA's consultant) and the weighting that each would carry. The task force members each assigned scores to each candidate publisher based on the evaluation criteria, and considering the written proposals, the oral presentations, and the responses to task force member questions, and intangible aspects. In the end, the task force voted unanimously to recommend the BMJ Group as the next JAMIA publisher. All task force members agreed that while there was higher risk for this strategy, due to the distance of London from the USA and various other factors, there was also the potential for a much higher reward for JAMIA both in recognition and impact (eg, close association with a highly regarded and widely circulated clinical journal), and in the potential for significantly enhanced financial returns to AMIA. The BMJ Group has been at the leading edge of innovation in electronic publishing and in promoting evidence-based medicine in the clinical arena. After receiving the final report and recommendation from the task force, the AMIA Board of Directors voted to invite the BMJ Group to serve as the next publisher of JAMIA, beginning with the January 2010 issue. The BMJ Group is a wholly owned subsidiary of the British Medical Association. It is based in BMA House on Tavistock Square in London. They have over 310 staff worldwide with 20 staff in the USA, including six physicians. They publish the British Medical Journal (BMJ) plus 30 specialty journals, for which the BMJ Group has 10 editors-in-chief based in North America. The AMIA Board decided that the BMJ Group would be more likely to extend JAMIA's (and AMIA's) brand, prestige, and impact through its high-profile presence and innovative publishing practices, as well as its market strength in the healthcare arena. The Board believed that JAMIA would thrive intellectually, enhance its electronic publishing functionality, and increase its relevance under the BMJ Group. The reviewers found several key points attractive in the BMJ Group's proposal, including: guaranteed income to AMIA for 2009 and beyond (with the potential for significant increases), direct association with the BMJ brand, change of access policies to maximize revenues, development of new revenue streams, transition costs covered by BMJ Group, plans to introduce a range of dynamic interactive features, and the opportunity to interact and share best practices with other international journal editors. The BMJ Group proposed a major focus on the journal's revenue growth, including enhanced direct marketing, streamlined management of the renewals process, the introduction of Web 2.0 features to build usage, and the nurturing of more extensive media contacts. The BMJ Group's proposal envisioned dynamic online JAMIA features. Potential examples included regular blogs, podcasts, and video features. Other online features might include topic collections, e-letters, and a JAMIA online community, as well as the bolstering of JAMIA's existing RSS feeds and data supplements. For authors and reviewers, the most prominent JAMIA enhancement will likely be the addition of Bench>Press, an online manuscript submission system developed by HighWire Press at Stanford University, in conjunction with BMJ Group. Compared with past JAMIA practices, Bench>Press offers a more accessible, available, and reliable method to submit manuscripts. Previously, authors were asked to submit their manuscript files via an FTP server, and to then correspond with the JAMIA Editorial Office by email. While this process provided a secure and relatively fast way to transmit files, a majority of JAMIA authors found the FTP process somewhat difficult. With Bench>Press, however, only an internet connection is needed and the submission instructions are very straightforward and easy to understand. The Bench>Press system, most importantly, will provide an online mechanism for authors to track the progress of submitted manuscripts through the JAMIA peer review process—something Editorial Office email did not support well. Another change authors will appreciate is the elimination of the PDF file requirement for submission. Bench>Press, as part of the submission process, will create a PDF version of the manuscript from the submitted text file. Authors will be responsible for checking the resulting PDF to make sure the conversion was successful. Authors will also be responsible for providing print-worthy figures before the manuscript enters the review process as opposed to after the manuscript has been accepted. Now, instead of including figures at the end of the manuscript file, each figure must be submitted as a separate file. The figure files will be uploaded along with the main manuscript file and checked for printability as part of the PDF conversion. If a figure is deemed not fit for print, the paper's corresponding author will receive notification along with a request to submit a figure file with better resolution. The submission will not enter the review process until authors have uploaded an image with better resolution. As part of the transition to the new publisher, and after much deliberation regarding the tradeoffs involved, AMIA is adjusting its policy regarding deposition of published articles from JAMIA into PubMed Central. As in the past, AMIA will retain exclusive rights to publish those submissions that can be copyrighted, but we will adopt a policy already pioneered by the BMJ Group whereby authors retain copyright on their articles (if copyrightable), and assign exclusive rights to publish and distribute the article to BMJ Group and to AMIA. This will allow authors, for example, to make copies of their work for non-commercial purposes (eg, for distribution to students in courses) without asking permission. The BMJ Group will provide assistance to assure that the final versions of all articles will be available in PubMed Central. The new publication process will continue to be compliant with all NIH (and certain other government or foundation funding agencies) requirements for open access 12 months after publication. Per the new publisher contract, those articles not covered under the NIH policy (or its equivalent) will not appear in PubMed Central until 36 months after initial JAMIA publication. To ensure a smooth and appropriate transition for authors, all accepted JAMIA articles that were initially submitted before January 1, 2010—that is, prior to the announcement of this change in policy—will be deposited in PubMed Central after 12 months (consistent with the prior policy). Of course, all articles will be available to AMIA members and to institutional and individual subscribers immediately via the JAMIA web site. The BMJ Group and AMIA will also provide an ‘unlocked’ option, whereby authors may pay a fee prior to publication to have their paper freely available, both in PubMed Central and on the JAMIA site, immediately upon first publication under a Creative Commons license. Details of the PubMed Central arrangements, as well as the ‘unlocked’ fee for 2010, and other ‘instructions to authors’, will be regularly updated on the new JAMIA website. This will include further details on authors' new rights with regard to the sharing of their accepted manuscripts on personal or institutional websites. Although these changes in policy will delay the release of certain JAMIA materials for free access, we will be working to attract more readers to JAMIA's own website, where we will be introducing new features and opportunities not available either in the print journal or on PubMed Central. We will be working closely with BMJ Group to monitor both the impact on revenues and the numbers of downloads, as well as JAMIA's impact factor. We are excited to share with readers some of the detailed plans for JAMIA's transition to a new and dynamic publisher, BMJ Group. The authors and the AMIA Board all believe that this is an important step forward to a better future for the journal. We will share more details with readers in the months ahead. We trust that AMIA's members, and all JAMIA readers and authors, will be pleased to see the evolution to a more timely and dynamic publication model that suitably leverages the new technologies, while continuing to provide the kind of quality articles and rigorous science that you have all come to expect of JAMIA. Comments to the AMIA leadership and to the JAMIA Editor are welcome as we move forward into the new world that lies ahead. AMIA would like to thank those individuals who served as members of the Publisher Selection Task Force: Dominik Aronsky, Morna Conway, Don Detmer, Sherrilynne Fuller, Karen Greenwood, Brian Haynes, Nancy Lorenzi, Randolph Miller, and William Tierney. None.

The JAMIA online publication site, www.jamia.org, is hosted by HighWire Press, which is a not-for-profit (and also, as HighWire states, not-for-loss) subsidiary of Stanford University. Recently, JAMIA and HighWire added a number of useful and interesting features for visitors to the JAMIA Web site, www.jamia.org. We describe these changes herein. To take advantage of these features, we remind subscribers to “activate” their JAMIA subscriptions on HighWire if they have not yet done so. Activation of a JAMIA subscription can be done by going to the JAMIA homepage and “clicking” on the “Subscriptions” link on the left-hand side of the homepage. The user will then be asked for their 10-digit “Customer Number.” For those who purchased an individual membership, the “Customer Number” was included with the payment confirmation letter. For AMIA members, the AMIA member number received will serve as the “Customer Number”; if the member number is less than 10-digits add an appropriate number of zeros to the beginning of the number.

OBJECTIVE: Clinical notes, typically written in natural language, often contain substructure that divides them into sections, such as "History of Present Illness" or "Family Medical History." The authors designed and evaluated an algorithm ("SecTag") to identify both labeled and unlabeled (implied) note section headers in "history and physical examination" documents ("H&P notes"). DESIGN: The SecTag algorithm uses a combination of natural language processing techniques, word variant recognition with spelling correction, terminology-based rules, and naive Bayesian scoring methods to identify note section headers. Eleven physicians evaluated SecTag's performance on 319 randomly chosen H&P notes. MEASUREMENTS: The primary outcomes were the algorithm's recall and precision in identifying all document sections and a predefined list of twenty-nine major sections. A secondary outcome was to evaluate the algorithm's ability to recognize the correct start and end boundaries of identified sections. RESULTS: The SecTag algorithm identified 16,036 total sections and 7,858 major sections. Physician evaluators classified 15,329 as true positives and identified 160 sections omitted by SecTag. The recall and precision of the SecTag algorithm were 99.0 and 95.6% for all sections, 98.6 and 96.2% for major sections, and 96.6 and 86.8% for unlabeled sections. The algorithm determined the correct starting and ending text boundaries for 94.8% of labeled sections and 85.9% of unlabeled sections. CONCLUSIONS: The SecTag algorithm accurately identified both labeled and unlabeled sections in history and physical documents. This type of algorithm may assist in natural language processing applications, such as clinical decision support systems or competency assessment for medical trainees.

The AMIA Annual Symposium is the premier forum for live presentation of scientific studies in biomedical informatics. Since its inception in 1976 as the Symposium on Computer Applications in Medical Care (SCAMC), the AMIA Symposium has showcased leading-edge informatics studies and promoted the exchange of ideas within a diverse community. Recognizing the importance of the symposium, as well as the reality of current academic promotion systems, the Scientific Program Committee (SPC) of the 2008 AMIA Annual Symposium developed a strategy to recommend outstanding AMIA manuscript submissions for possible publication in JAMIA. Since MEDLINE® has for years indexed the articles that appear in the AMIA Symposium, JAMIA has traditionally treated those works as “already published.” Thus, in order for a paper published in the AMIA Proceedings to merit consideration as a JAMIA submission, JAMIA has required authors to add new methods, and produce substantially different results that lead to new insights. The intent of the current new initiative was to motivate authors to submit their best work and have it presented at the Annual Symposium, while also giving selected authors the opportunity to publish nonduplicative work in a scientific journal. The key enabling factor was early review of AMIA Symposium submissions that allowed us to invite authors of outstanding AMIA papers to convert them to JAMIA submissions directly, while substituting abstracts of less than 400 words to replace the selected manuscripts' original submissions in the AMIA Proceedings. In the first year of the initiative, we implemented a pilot strategy. The 2008 AMIA Scientific Program Committee nominated only a very small number of manuscripts for consideration in JAMIA. From twenty-two nominated manuscripts, the JAMIA Editorial Office selected nine for full review as potential JAMIA submissions. The SPC invited the authors to revise their manuscripts according to recommendations from AMIA reviewers, and to extend the text wherever necessary. The authors were free to follow the recommendation and submit to JAMIA, or to decide to leave their AMIA papers “as is” in the AMIA Proceedings and not submit to JAMIA. All of the papers appeared as regular presentations at the AMIA Meeting. Those papers that did not pass JAMIA peer review still appeared in the AMIA Proceedings as 400-word abstracts, and authors can still submit the corresponding full-length papers elsewhere if they choose to do so. We include in this issue a set of four AMIA submissions that passed full, rigorous JAMIA peer review after expansion as “selected AMIA manuscripts”. By coincidence, all four relate to information retrieval and natural language processing applications: Kilicoglu et al.1 demonstrated the feasibility of utilizing machine learning algorithms to automate the process of retrieving scientifically rigorous clinical research evidence from the literature, and discuss the potential for clinicians to benefit from this application. Lu et al.2 showed that document retrieval by a simple term-weighting approach (TF-IDF3) outperformed retrieval based on sentence-level co-occurrence in a given set of queries for MEDLINE. The authors advocate for the use of this approach in PubMed®. The two manuscripts related to natural language processing include: Uzuner et al.4 who extended the rule-based NegEx algorithm5 to cover alter-association assertions in an Extended NegEx (ENegEx) system and compared it to a Statistical Assertion Classifier (StAC). The authors showed that StAC models developed on a training set of discharge summaries outperform ENegEx when applied to a previously unseen (test) set of radiology reports. Xu et al.6 showed that a clustering-based method outperforms manual annotation in the task of building sense inventories of clinical abbreviations for randomly selected samples of hospital admission notes. Three other manuscripts originally submitted to the AMIA Symposium and later extended for JAMIA will appear in the next JAMIA issue7–9 and are currently available on JAMIA's Web site as publish-ahead-of-print “PrePrints.” The remaining two manuscripts are currently undergoing revision and their final JAMIA status will be determined at a later date. The SPC for the 2009 AMIA Annual Symposium has committed to continue and expand support for this initiative. We expect an even larger number of outstanding submissions to the AMIA Symposium by the deadline of March 13, 2009. The SPC will scale up the pilot strategy adopted in 2008 to promote a substantial increase in the number of manuscripts that are invited for potential publication in several biomedical informatics journals. Annual symposium attendees, biomedical informatics journal readers, and especially authors will benefit from the flexibility of this strategy, which constitutes a small but important step towards the goal of rewarding excellence in the field of biomedical informatics.

OBJECTIVE: Evaluations of individual terminology systems should be driven in part by the intended usages of such systems. Clinical interface terminologies support interactions between healthcare providers and computer-based applications. They aid practitioners in converting clinical "free text" thoughts into the structured, formal data representations used internally by application programs. Interface terminologies also serve the important role of presenting existing stored, encoded data to end users in human-understandable and actionable formats. The authors present a model for evaluating functional utility of interface terminologies based on these intended uses. DESIGN: Specific parameters defined in the manuscript comprise the metrics for the evaluation model. MEASUREMENTS: Parameters include concept accuracy, term expressivity, degree of semantic consistency for term construction and selection, adequacy of assertional knowledge supporting concepts, degree of complexity of pre-coordinated concepts, and the "human readability" of the terminology. The fundamental metric is how well the interface terminology performs in supporting correct, complete, and efficient data encoding or review by humans. RESULTS: Authors provide examples demonstrating performance of the proposed evaluation model in selected instances. CONCLUSION: A formal evaluation model will permit investigators to evaluate interface terminologies using a consistent and principled approach. Terminology developers and evaluators can apply the proposed model to identify areas for improving interface terminologies.

OBJECTIVE: Hyperglycemia worsens clinical outcomes in critically ill patients. Precise glycemia control using intravenous insulin improves outcomes. To determine if we could improve glycemia control over a previous paper-based, manual protocol, authors implemented, in a surgical intensive care unit (SICU), an intravenous insulin protocol integrated into a care provider order entry (CPOE) system. DESIGN: Retrospective before-after study of consecutive adult patients admitted to a SICU during pre (manual protocol, 32 days) and post (computer-based protocol, 49 days) periods. MEASUREMENTS: Percentage of glucose readings in ideal range of 70-109 mg/dl, and minutes spent in ideal range of control during the first 5 days of SICU stay. RESULTS: The computer-based protocol reduced time from first glucose measurement to initiation of insulin protocol, improved the percentage of all SICU glucose readings in the ideal range, and improved control in patients on IV insulin for > or =24 hours. Hypoglycemia (<40 mg/dl) was rare in both groups. CONCLUSION: The CPOE-based intravenous insulin protocol improved glycemia control in SICU patients compared to a previous manual protocol, and reduced time to insulin therapy initiation. Integrating a computer-based insulin protocol into a CPOE system achieved efficient, safe, and effective glycemia control in SICU patients.

Background Up to 38% of inpatient medication errors occur at the administration stage. Although they reduce prescribing errors, computerized provider order entry (CPOE) systems do not prevent administration errors or timing discrepancies. This study determined the degree to which CPOE medication orders matched actual dose administration times. METHODS At a 658-bed academic hospital with CPOE but lacking electronic medication administration charting, authors randomly selected adult patients with eligible medication orders from historical 1999-2003 CPOE log files. Retrospective manual chart audits compared expected (from CPOE) and actual timing of medication administrations. Outcomes included: dose omissions, median lag times between ordered and charted administrations, unauthorized doses, wrong dose errors, and the rate of nurses' medication schedule shifting. RESULTS Dose omissions occurred in 756 of 6019 (12.6%) audited administration opportunities; only 313 of the omissions (5.2% of opportunities) were unexplained. Wrong doses and unexpected doses occurred for 0.1% and 0.7% of opportunities, respectively. Median lag from expected first dose to actual charted administration time was 27 minutes (IQR 0-127). Nursing staff shifted from ordered to alternate administration schedules for 10.7% of regularly scheduled recurring medication orders. Chart review identified reasons for dose omissions, delays, and dose shifting. CONCLUSION Inpatient CPOE orders are legible and conveyed electronically to nurses and the pharmacy. Nonetheless, ward-based medication administrations do not consistently occur as ordered. Medication administration discrepancies are likely to persist even after implementing CPOE and bar-coded medication administration unless recommended interventions are made to address issues such as determining the true urgency of medication administration, avoiding overlapping duplicative medication orders, and developing a safe means for shifting dosing schedules.

The case report by Welker and McCue in the current issue of JAMIA1 raises a number of important ethical concerns, both directly and indirectly. First is the longstanding debate, discussed by Welker and McCue, about what constitutes authorship. Some recent reports2–6 suggest replacing “authorship” with “contributorship.” Under such a model, submitted manuscripts would list all individuals contributing to a publication, along with careful documentation of each individual's actual contribution(s) to the work. This model would provide for only a small number of “key” project members, such as the principal investigator and individual(s) who both contribute in major ways and who also write large segments of the manuscript, to have “author” status. Even short of authorship, there exists an ethical impetus to acknowledge important contributors to published works. One might address the dilemma presented in the case report by Welker and McCue simply by placing in the Acknowledgments section of a paper a citation to the original (local) developers of the software tools used. Similarly, the Acknowledgments section might also list the agencies that funded local development of software tools used. Mechanisms exist for developers of biomedical software tools to describe their software, per se, in the literature. For example, JAMIA publishes Methods papers describing new approaches to a problem; Application of Information Technology papers describing new applications with an emphasis on general “lessons learned;” and Implementation Briefs, which provide helpful hints about a successful to approach an important common problem—even if such a solution is not novel. Analogous venues exist for describing software tools in other journals. Stead et al. pointed out in the first issue of JAMIA that the stage of a software project's development and deployment determine the level of evaluation required for publication.7 Contrary to the position taken by Welker and McCue, the larger ethical issues surrounding use of intellectual property developed by others are not straightforward. Complex ownership and contributor relationships are rarely as simple as not claiming authorship credit if one “provided the pencil to Shakespeare that he used to write Hamlet,” or not claiming credit if one provided a piece of laboratory apparatus to a student for an experiment.1 Researchers developing novel analytical methods or novel software tools as the intellectual products of externally sponsored research are in general expected to share the results of their work by describing it in the peer-reviewed literature. However, unless pre-specified by the funding agency in a grant or contract, the obligation to share does not automatically extend to placing ownership of research-related intellectual property in the public domain, even though it is a good practice to do so. Placing original works in the public domain is one of many options available to developers of intellectual property, as discussed below. It would be unusual for a separate group of individuals to immediately claim the right to use or apply the “new” research results of an earlier group, prior to the time of first publication by the original developers. However, making the results of research accessible to others, without restrictions, may mediate against ownership claims. How the release of the software tool occurs in part determines the obligations of others in using the tool. For example, in the case reported by Welker and McCue, consider a scenario in which the original software developers had released their tool for general use under a proviso (prospectively issued) that the tool developers made the release with the goal of identifying collaborators who could apply the tool to demonstrate its effectiveness for research, in order to publish a joint evaluation of the tool. Under such circumstances, at least the first “outside” group to apply the tool to their own research would be obligated to discuss, before any project was undertaken or paper written, issues of future authorship with the original software tool developers. Such discussions typically determine, prospectively, which individuals receive designation as potential authors, so that those persons can fulfill required obligations of authorship. However, in a different scenario, in which the tool developers issued no such request prospectively, and the tool was placed in general use without restrictions, it would be unreasonable to expect, several years later, that any application of the tool would encumber co-authorship rights for the original developers of the tool. It would be especially unreasonable if an authorship request were made “post facto”, after a study using the tool had been completed and “written up” by another group. In between these clear-cut extremes lie gray zones where it may not be certain what the obligations of the second group are to the first group. There are other ethical and legal issues related to the case report by Welker and McCue. As described, the process of using a “Clinical Viewer” tool to detect, in real time, patients eligible for a study, and to then enroll patients in a study, requires adherence with important HIPAA (Health Information Privacy and Accountability Act) regulations, and also requires prior institutional review board (IRB) approval for research on human subjects. Presumably, such safeguards were in place at the authors' institution. A broader discussion of legal and ethical issues related to the use of clinical software systems has occurred over decades.8,9 In the realm of bioinformatics (computational applications related to “molecular medicine”) software development, somewhat different paradigms and expectations have evolved. A common expectation of bioinformatics journals and reviewers is that publication of a software tool will be contingent upon its open availability, along with the availability of datasets described in the publication that are used to demonstrate the functionality of the tool. In some cases this is explicitly stated in instructions to authors. For example, Oxford Bioinformatics states “Software or data must be freely available to non-commercial users. Availability must be clearly stated in the article. Authors must also ensure that the software is available for a full TWO YEARS following publication. Web services should not require mandatory registration by the user.10” In summary, the intellectual property issues associated with software development, and related issues of authorship are not simple. The case report by Welker and McCue helps to illustrate this point.

In the current issue of JAMIA, Kawamoto and Lobach1 propose a software framework intended to facilitate widespread, effective, clinical decision support. The proposed framework embraces a service-oriented-architecture (SOA) approach. Service-oriented-architecture is a philosophy of design described as “the software equivalent of Lego bricks,”2 where a toolset of mix-and-match units (“services”), each performing a well-defined task, can reside on different machines (including geographically separated ones), ready to be used when needed. The most widespread implementations of SOA involve the use of Web services, where a given computational resource/service can be invoked by a remote machine via messages composed in XML and sent over HTTP, so that they can operate across firewalls. Think of a Web service, in its simplest form, as a subroutine that can be called over the Internet. Thanks to success stories such as Amazon.com,3 and at software giants such as SAP,4,5 the status of SOA in the business information technology (IT) domain has risen meteorically. The Kawamoto and Lobach paper1 reiterates the following potential benefits of SOA: Simpler software design and implementation, by decomposing complex problems into smaller, more manageable ones. Improved software reusability through enhanced reuse of existing IT resources. Improved adaptability to changing business requirements. Cost savings consequent to the above benefits. Some IT articles, however, present a “caveat emptor” skepticism about SOA.6,7 One must carefully inspect the claimed benefits of SOA closely, examine its potential drawbacks, and take a balanced approach to any proposed new SOA application while taking into consideration past lessons from business IT. In software design, simplification through problem decomposition into semi-independent units (subroutines, classes) is an important, incontrovertible principle. In SOA, where the units (services) lead a relatively autonomous existence (e.g., units are free to reside on physically separate hardware), decomposing a software solution into individual services makes sense only in two circumstances: When a particular service can provide independent value to the remote caller. When physical separation of services on multiple machines provides quantifiable scalability and performance benefits through hardware parallelism that more than offset the extra overhead of intermachine communication. Typically, the latter economies occur when the services are used mostly internally, and even then, achieving scalability may require upgrades to internal network infrastructure to minimize this overhead. When a user of Amazon.com asks for information on a book, for example, several dozen independent services compose individual parts of the returned Web page, such as product information, book rank, similar titles, and customer reviews. Newman7 points out that reuse of an existing resource requires that the resource was designed to be reusable. Designing, or redesigning, for reusability involves one or more of the following steps: Rigorously examining the assumptions implicit in an existing resource's design and determining which of these do not hold in a different circumstance. For example, consider the situation where a service's output must eventually be converted into human-readable text, as with Amazon's Web pages, or the recommendation system for preventive clinical procedures postulated in this issue's paper. One issue to be determined is whether the service should have a transnational reach, so that output is not restricted to the English language alone. Creating localizable software (i.e., software that can be adapted for different cultures without having to completely reprogram it) involves, among other things, isolating the textual elements that are to be shown to the user in “resource files” and referring to them symbolically. While software tools to facilitate localization are quite mature (e.g., Microsoft Visual Studio Team Suite®), reworking an existing software resource to make it localizable is inherently a painstaking and error-prone task. Identifying a general underlying problem amenable to creation of a general solution. Deciding to make something reusable often requires changing an organization's business model. Linquist2 cites ProCard, a credit-card service provider whose strengths resided in its software suite. ProCard determined that converting its suite into independently accessible services was worthwhile only if it transformed itself from a competitor of Visa and MasterCard to a provider of software services to its much larger rivals. Deciding whether the extra effort needed to re-engineer and/or generalize the solution is justified in the present situation, given constraints such as urgency and limited developer resources. The caveats here are that, as in great architecture and literature, simplicity is an elusive entity whose achievement is as much art as science: that is why it is so uncommon. Regarding reusability, Newman7 estimates that it generally takes at least three iterations to design a reusable software unit (class, library, service) well. Individuals capable of abstracting problems and devising elegant, simple, and general solutions tend to be both rare and expensive. It might be more accurate to state that SOA is a strategic, long-term objective rather than a tactical, short-term one. As with other system design approaches—e.g., metadata-driven software architectures—developers must invest a significant amount of effort in framework-building, which takes time, resources, and human expertise, before anticipated payoffs in reduced costs and greater adaptability are realized. For both Amazon and SAP, the time investment was at least four years. For a service that can provide independent value, the need to adhere to existing standards (when the service may be called from outside the organization, and must be interoperable with services designed by others) often necessitates extra effort that would not otherwise be required. Here the challenges concern the semantics of the data that must be passed to and from the service and how it must be represented, rather than the nature of the XML plumbing: modern software development environments mostly shield you from the latter and allow you to concentrate on the former. While the paper by Kawamoto and Lobach emphasizes the use of a standard, the HL7 Service Functional Model (SFM) Specification for Decision Support Service8 we note that this standard is in its early stages, with version 0.85 posted June 19, 2006 and version 1.0 posted July 23, 2006. In addition, we note that Dr. Kawamoto is a member of the HL7 SOA SIG, and was project leader on the HL7 Decision Support Service effort. Considerable experience will be needed to determine whether the specification is sufficiently comprehensive to meet its objectives and to what extent it will need to be revised. Amazon was fortunate in that they were the sole standards-setter for the family of services that they offer for external use. In an area where consensus is not yet present, however, the lead time required for consensus to be achieved must be factored in. In the medical domain, where HL7 version 3's status is still not official (for reasons that are not entirely technical but beyond the scope of this editorial), the reasons and process for implementing SOA must be similar to the ones in IT elsewhere. That is: The case for SOA must be made from a business perspective. The initial implementations of SOA must focus on internal use (where the services are used primarily over the organization's Intranet) so that one can worry less about nascent and immature standards and focus on solving specific problems. SOA must be understood to be a long-term, strategic approach that is not always applicable. In particular, as stated in Lindquist,2 if the reusability quotient of a given problem is low, a “one-off” non-SOA solution turns out to be more direct and faster to create. The paper by Kawamoto and Lobach1 mentions “service discovery” as one of the aspects of SOA. The idea of service discovery is that a service contains descriptive information about itself that would allow a remote automated software agent to determine whether a service exists on the Internet that would address a particular need. The HL7 specification discussed above provides for keywords that would assist the search for a particular service. While “semantic Web” researchers have made much of the potential of service discovery, this is currently far removed from reality. When carefully considered, service discovery is an extremely difficult problem to solve. If we regard the collection of biomedical services as a vast Internet-accessible subroutine library, determining whether a service is appropriate for a specific task would require considerable expert human intervention. Software developers who have to work with giant development frameworks such as Java® and Microsoft.NET®, which contain tens of thousands of subroutines, have to deal regularly with the same problem in a non–Web services setting. As often as not, browsing vendor documentation is insufficient, forcing the developer to use a Web search engine such as Google® and browse resources such as developer group forums and blogs. At present, the large Web service providers simply provide extensive documentation on each service, which also includes programming examples as well as case histories of successful use. The best service providers continually solicit feedback on the quality of their documentation in order to improve it. Clinicians, health care organizations, consumer groups, and informatics developers regard clinical decision support as a key method to address the inefficiencies and errors documented to occur in busy clinical practices.9,10 However, many health care organizations now see decision support as a selective means to provide better care to their patients in a manner that distinguishes them from their competitors. For the reasons listed above, SOA within a local environment may greatly facilitate such clinical decision support. However, there are potentially severe concerns if an organization “outsources” its decision support to potentially imperfect external agencies. The manner in which SOA is implemented, as noted by Kawamoto and Lobach, is as a series of “black boxes” that, given an input, produce an output. In general, SOA services are not “licensed practitioners,” so that legally, the patient's health care provider (clinician or institution) is responsible for overriding any erroneous advice provided by an SOA service. The latter circumstances may in and of themselves inhibit reliance on external clinical SOA services. If one were to ask a commercial software vendor, a pharmaceutical company, or a health care-related agency to provide a “medication dosing service” at a regional or national level, who would trust the service to provide uniformly correct answers? If a hypothetical SOA medication dosing service only took age into consideration, and not weight, it could not be used in pediatrics or for dosing certain medications, such as aminoglycoside antibiotics, in adults. Even if a dosing service took weight and age into consideration, but not gestational age or height, it still could not be used for neonatal applications or for situations in which doses are based on body surface area (e.g., chemotherapy). Dosing also depends on renal and hepatic function. Finally, dosing is often diagnosis-dependent. The same adult patient, at a given age, weight, height, and level of renal function, would require far higher doses of a “correct” beta lactam antibiotic when treating bacterial endocarditis in an inpatient setting than would be required to treat that patient's community-acquired pneumonia in an outpatient setting. How much information would a “medication dosing service” require as input in order to give proper advice in all settings, and how complex would the output have to be to cover a myriad of potential patient characteristics? In the best commercial drug databases on the market today, the algorithmic and data-structure sophistication for such a dosing service does not yet exist; the best that these systems do is reproduce the medication vendor's package insert text. Converting such an existing imperfect solution to a Web service would not be satisfactory. The logistics of responsibly implementing long-distance clinical SOA services may be overwhelming. The “black box” model for SOA services assumes relatively minimal input and output. Consider, however, the circumstance whereby the maintainers of a regional or national SOA clinical decision support service discover that the advice provided by the service has been faulty over the past 24 hours due to a “bug” introduced into the software (or the knowledge base) of the service. What mechanisms would exist that would enable the SOA provider to contact all care providers who had relied upon the service to provide advice for patients, and to then determine which patients' ordered regimens would need to be changed? To do so would require an “SOA transaction identifier” that at least goes back to identifying the institution requesting the service, and possibly even to the level of identifying the patient (at the SOA level). This raises the possibility whereby enough information about a patient must be transmitted to an SOA service to obtain advice that HIPAA privacy rules are violated. If SOA services are vended commercially (as might be the case in some future scenario involving the authors' commercialization efforts), one must consider how to charge the users of the services. Some of Amazon's services, e.g., product data (prices, images, customer reviews), are free, while other services are priced: e.g., the Historical Pricing service, which provides access to more than three years of Amazon's sales data for any item, has a fee of $249/month for up to 60,000 requests per month. The legal and ethical issues related to how such a commercial SOA decision support system might operate across state, and possibly national borders, have yet to be addressed in a definitive and thoughtful manner.

As the Editors of leading international biomedical informatics journals, the authors report on a recent pattern of improper manuscript submissions to journals in our field. As a guide for future authors, we describe ethical and pragmatic issues related to submitting work for peer-reviewed journal publication. We propose a coordinated approach to the problem that our respective journals will follow. This editorial is being jointly published in the following journals represented by the authors:Computer Methods and Programs in Biomedicine, International Journal of Medical Informatics, Journal of Biomedical Informatics, Journal of the American Medical Informatics Association, and Methods of Information in Medicine. As editors, we have collectively experienced at least one of the following occurrences recently: (1)Concurrent duplicate submissions: The same set of authors submits essentially identical manuscripts to two separate journals concurrently, without disclosure to the editorial staffs of either. The authors may mistakenly believe that it is permissible to do so because the respective journals have minimally overlapping audiences. (2)Serial unaltered submissions (“journal shopping”): Authors submit a manuscript to one biomedical informatics journal, and, after peer review, it is not accepted for publication, and a critique is provided. The authors do not make any of the changes suggested by the previous review and instead submit the unchanged manuscript immediately to a second journal, without disclosing the existence or results of the previous review by the first journal. (3)Serial minimally altered republication: Authors publish a preliminary manuscript as part of conference proceedings. Mistakenly believing that conference publications do not count as “official” publications (of note, several informatics conference proceedings, such as MEDINFO, MIE, and the AMIA Fall Symposium, are indexed in MEDLINE), the authors later submit the same work, with minimal alteration or expansion, to a peer-reviewed journal for publication. (4)Self-plagiarism1: Authors, mistakenly believing that any text that they have written is “theirs,” submit a new manuscript for publication in a different, peer-reviewed journal and include major sections (paragraphs or larger) of the previous peer-reviewed publication that they authored—and do so without proper attribution to the original source or without obtaining permission from the copyright holder. (5)Nondisclosure of conflict of interest by one or more of the authors: Authors with a financial interest related to the scientific content of the paper fail to disclose this information in a cover letter to the editor at the time of submission or in the acknowledgment section of the manuscript prior to the time of publication.

OBJECTIVE: In the context of an inpatient care provider order entry (CPOE) system, to evaluate the impact of a decision support tool on integration of cardiology "best of care" order sets into clinicians' admission workflow, and on quality measures for the management of acute myocardial infarction (AMI) patients. DESIGN: A before-and-after study of physician orders evaluated (1) per-patient use rates of standardized acute coronary syndrome (ACS) order set and (2) patient-level compliance with two individual recommendations: early aspirin ordering and beta-blocker ordering. MEASUREMENTS: The effectiveness of the intervention was evaluated for (1) all patients with ACS (suspected for AMI at the time of admission) (N = 540) and (2) the subset of the ACS patients with confirmed discharge diagnosis of AMI (n = 180) who comprise the recommended target population who should receive aspirin and/or beta-blockers. Compliance rates for use of the ACS order set, aspirin ordering, and beta-blocker ordering were calculated as the percentages of patients who had each action performed within 24 hours of admission. RESULTS: For all ACS admissions, the decision support tool significantly increased use of the ACS order set (p = 0.009). Use of the ACS order set led, within the first 24 hours of hospitalization, to a significant increase in the number of patients who received aspirin (p = 0.001) and a nonsignificant increase in the number of patients who received beta-blockers (p = 0.07). Results for confirmed AMI cases demonstrated similar increases, but did not reach statistical significance. CONCLUSION: The decision support tool increased optional use of the ACS order set, but room for additional improvement exists.

Previous investigators have defined clinical interface terminology as a systematic collection of health care-related phrases (terms) that supports clinicians' entry of patient-related information into computer programs, such as clinical "note capture" and decision support tools. Interface terminologies also can facilitate display of computer-stored patient information to clinician-users. Interface terminologies "interface" between clinicians' own unfettered, colloquial conceptualizations of patient descriptors and the more structured, coded internal data elements used by specific health care application programs. The intended uses of a terminology determine its conceptual underpinnings, structure, and content. As a result, the desiderata for interface terminologies differ from desiderata for health care-related terminologies used for storage (e.g., SNOMED-CT), information retrieval (e.g., MeSH), and classification (e.g., ICD9-CM). Necessary but not sufficient attributes for an interface terminology include adequate synonym coverage, presence of relevant assertional knowledge, and a balance between pre- and post-coordination. To place interface terminologies in context, this article reviews historical goals and challenges of clinical terminology development in general and then focuses on the unique features of interface terminologies.

Electronic prescribing (e-prescribing) systems can provide computer-based support for the creation, transmission, dispensing, and monitoring of pharmacological therapies. In the United States and other countries, such systems have been documented, under certain conditions, to increase the safety and quality of patient care.1–5 The authors applaud the initial efforts of Teich and colleagues in the Joint Clinical Decision Support Workgroup (Joint CDS WG) to outline e-prescribing desiderata, as reported in this issue of JAMIA by Teich et al.6 Their article is published as an endorsed policy of the American Medical Informatics Association (AMIA). Previously, Bell et al. published an excellent list of desiderata for outpatient e-prescribing and sorted the desiderata into functional categories.7 Subsequently, Wang et al. surveyed e-prescribing vendor systems to determine that existing systems on average met only half the desiderata, with none exceeding 64% fulfillment.8 The recommendations outlined in the tables of the Joint CDS WG provide a useful point of departure for future discussions. Of note, the Joint CDS WG guidelines were developed as a “commissioned work” with externally determined foci, time limitations, and priorities, so that those guidelines do not fully cover all relevant areas. The Joint CDS WG document therefore represents an important first step in an evolving approach to a complex set of problems. The Joint CDS WG recommendations present a scenario of how e-prescribing features might be rolled out. The authors of this commentary would like to supplement, from what we believe is a broader perspective, the focused set of Joint CDS WG recommendations. The Joint CDS WG proposal has several strengths, including the recommendations that the United States should develop and promote shareable standards for e-prescribing and related decision support systems, a consensus should be developed on how to implement and evaluate decision support systems, and certain organizations, (such as the Office of the National Coordinator for Health Information Technology, the Agency for Healthcare Research and Quality, the U.S. Food and Drug Administration (FDA), the National Library of Medicine, AMIA, the e-Health Initiative, and the Health Information and Management Systems Society) should take leadership roles in the e-prescribing efforts. The authors note that developers and implementers should consider the variability that currently exists among users, clinical settings, information systems, and environments when determining how and when to install and support an e-prescribing system. For example, even when clinical systems provide net benefits to an institution, the implementation of electronic systems to improve the quality of care can introduce unwanted, potentially harmful side effects that must be detected, monitored, and addressed.9,10 It is therefore important to consider the potential adverse effects of e-prescribing implementation. Electronic prescribing systems represent only one genre of electronic health record system activity (others include departmental pharmacy, radiology, and laboratory systems and systems for record keeping, ordering, results display, monitoring, and decision support). Because the current state of the art for complex, comprehensive electronic health record systems is immature,11 there is not yet a scientific basis for selecting among the many potential courses of action related to implementation and use of e-prescribing systems. It is the authors' opinion that human (end-user) factors and electronic information interchanges among e-prescribing and other clinical systems play critically important roles in determining the success or failure of e-prescribing systems. These considerations should be combined with the Joint CDS WG suggestions when making implementation decisions. Several articles in the current issue of JAMIA illustrate how intricate and difficult it is to implement and evaluate such systems. There are few operational systems in place that have documented the success or failure of e-prescribing guidelines outlined. The authors note that e-prescribing systems alone may not suffice; more comprehensive electronic health systems may be required to address the needs of both healthcare facilities and individual practitioners. Clinicians should be wary of developing a false sense of security and unrealistic expectations based on use of e-prescribing applications alone, when more complex systems may be required. To reap the benefits of several decades of dedicated work by biomedical informaticians, commercial vendors, and health care providers (institutions and individuals), a responsible approach to e-prescribing must be advocated. All parties with a stake in e-prescribing must develop a common, overarching framework for its development and dissemination. The remainder of this commentary examines the environmental factors, technical factors, and strategic factors relevant to e-prescribing before concluding with a recommended framework that builds on and supplements the Joint CDS WG e-prescribing recommendations. Environmental factors relevant to e-prescribing include individual practitioners' specialties and roles; the variety of practice settings in which care is delivered in the United States; the standard of care in the clinical community; and end-user constraints imposed by human limitations in knowledge, habits, and work flows. Pragmatically, clinicians' individual practice circumstances should determine their access to, choice of, and use of e-prescribing technology. Currently in the United States, the status of e-prescribing systems varies by geographic regions and by federal, state, or local governmental jurisdictions; practice setting/type; and commercial vendor application. As a result, there are widely varying e-prescribing adoption rates. It might be argued that in today's health care environment, a rural general practitioner who manages inpatients in the morning, outpatients in a small office in the afternoon, and who makes house calls as needed cannot and should not use the same e-prescribing system in each setting, even though in the future one system should suffice. Today's systems with one underlying knowledge base cannot easily switch between inpatient and outpatient formularies (which differ significantly). The rich information environment of hospital-based electronic medical record and computerized provider order entry (CPOE) systems is not easily replicated in outpatients' homes, even with remote wireless connectivity. Large academic medical centers have the resources (adequate teams of talented informaticians, available expert clinicians, and an annual budget of millions of dollars to support their work) to develop or purchase, customize, roll out, and evolve state-of-the-art e-prescribing systems. By contrast, both solo providers and small rural hospitals have limited access to informatics expertise, and little time or money to develop or install complex systems.12,13 In the inpatient setting, when a clinician orders a medication, there is typically only one inpatient pharmacy system through which the order will be processed, and it is the same system for all providers and all orders. In the outpatient setting, patients typically receive multiple prescriptions from multiple care providers and may fill them at different pharmacies. Each retail pharmacy store (or pharmacy chain) may have its own software system that provides various levels of alerts regarding doses and drug interactions to pharmacists as they fill prescriptions, but the same prescription taken to different pharmacies will generate different alerts. Electronic connectivity is rare between free-standing outpatient pharmacies and the hospital or clinic-based, patient information–rich practice settings where providers generate prescriptions. Most connectivity that exists takes the form of fax machines, which, it is hoped, produce legible prescriptions but which still do not preclude transcription (and other) errors. For example, it would not be uncommon for a physician to write, “warfarin 5 mg one tablet by mouth daily” and the pharmacist to dispense (due to a temporary shortage) 2.5 mg tablets, with the instruction “take two tablets daily.” When in the following week, the physician receives a subtherapeutic international normalized ratio (INR) blood test result for the patient (indicating that the dose should be increased), a phone call to the patient's home to “take one and one-half pills daily” may have disastrous consequences. The patient would take a decreased dose (3.75 mg) instead of the physician's intended dose (7.5 mg) due to failed communication among systems, providers, and the patient. “Closed loop” e-prescribing feedback that matches clinicians' orders with pharmacy dispensing annotations requires a bidirectional interface from prescribing site to the dispensing pharmacy. What is required is that the systems “match up” the dispensing record with the prescribing order in a manner in which any clinician reviewing the patient's chart could easily determine what was ordered and how it was dispensed, so as to avoid the previous scenario. The few bidirectional interfaces now in existence often generate a request for the attending physician's countersignature whenever the pharmacist changes the dispensing record. The latter model is unworkable in terms of introducing an unnecessary and often confusing extra burden on the physician, who may not understand which medication order is being changed for what dispensing reason if multiple changes are made. Clinicians who are told to use an e-prescribing system cannot be expected to do so if using the system compromises the clinicians' existing standard of care for patients. Weight-based dosing represents the standard of care for many medications and ages in the pediatric population. A 1998 article in Pediatrics titled “Prevention of Medication Errors in the Pediatric Inpatient Setting” listed “confirm that the patient's weight is correct for weight-based dosages” as its first recommendation under “medication ordering to reduce errors.”14 In a related article in 2004, Kaushal et al. cited error-prone behaviors in clinicians' manual calculations of weight-based pediatric medication dosages.15 The authors believe that weight-based dose calculations for pediatric patients is an established standard of practice in the community that should be adopted within e-prescribing systems immediately (i.e., by 2006), even though many current vendor pharmacy and CPOE products poorly support pediatric dosing (and especially dosing in premature neonates). A number of academic centers have demonstrated that pediatric dosing can be done as part of electronic prescribing. Those centers have developed, deployed, and evaluated reliable pediatric and neonatal dosing systems over the In the current issue of et present a of a that potentially in part by use of an inpatient e-prescribing system. the of complex interactions among system and environmental factors that can to whenever and feedback Systems that clinician by not all relevant information for decision making into one place the of the clinician must still the record (or a clinical results as as with an e-prescribing system the result can be more work and for the clinician as as more to by important a or so with can a of errors. The in this issue by and provides a of how systems that might be can effects when in the clinical environment due to a of failure of to the system clinical of the environment, and of various among different electronic systems, among clinicians, or between and systems can to errors. must be to the environment as as to the systems being In the current issue of the by et al. the factors that clinician with and that environmental factors can play roles in or to use of between and using the not with the implementation of recommended of clinical recommendation and interface the number of at a strategic of the of into and the to document system and receive In previous of et al. the human factors required for implementation of clinical and and argued that the important implementation are factors relevant to e-prescribing systems include the information of a the for system the development for the system the to which clinician in feedback at the implementation and use the electronic interface of the system to other electronic the interface of the the of a combined who and system the of local to have into system and the to changes within to not or and and for quality both and developers of such systems. an base exists in the under e-prescribing systems can improve patient safety and quality of the under which such cannot be easily Most the of CPOE and electronic medical record systems have from academic medical centers that the resources and both informatics and to and clinical decision support the information knowledge underlying commercial CPOE and pharmacy system vendor products are typically by commercial (such as and and to CPOE The quality and of pharmacy information have been and their has at been into The efforts reported by et are of this the success reported by that such when can be The and safety of e-prescribing systems on their underlying pharmacy information the of the and the circumstances of software and system implementation. In the systems available to the of hospitals and in the United States were developed by commercial vendors, systems typically the state of the art in the academic centers by several For example, that vendor systems are in the of pediatric prescribing. The authors are of a number of of hospitals pharmacy systems and or the pediatric prescribing due to of by the systems. of one commercial drug information system in an e-prescribing (such as dose of in and for and that use of the system local was The of and the of with many commercial vendor systems (and are not to the Most commercial pharmacy information systems generate a (and number of drug et al. a for all based on a failure of the to how often interactions in patient and the of those when they As et al. note, often at the physician to in the et al. the of resources that must be dedicated to and the commercial drug information that must be at medical that such systems. The of at each local medical the are both and at local is required at system desiderata, including use of were in the of Bell et the Joint CDS WG et and in the for system developers outlined by and In to desiderata, it is to consider and when e-prescribing systems. For example, information should be from only when the information will be for important decisions. the should be only from the individual to the correct a clinician in patient or laboratory results so that the information can be as or on a prescription may be the e-prescribing system should decision support that laboratory results or if to provide dosing recommendations and The to e-prescribing system reviewing alerts and taken is for quality in As by and in this issue of such as may the of for use by decision support systems and may be to of e-prescribing of factors relevant to e-prescribing include of or how to e-prescribing systems through published related to the of e-prescribing system (such as governmental and Joint on of Healthcare related to and and such as development of standards monitoring of an e-prescribing system over time on the quality and of the knowledge base underlying the e-prescribing system at any point in the quality and of the software system the knowledge base to a patient's clinical for prescribing the and for of both the knowledge base and the The Joint CDS WG that drug information should be if a number of such are of one and if their knowledge and terms of quality of and over they might not to the quality of drug information systems. The of for e-prescribing systems is an but we cannot how each of the to by the Joint CDS WG will to it is important to of the underlying drug information knowledge base from of the related software cannot for quality drug and an quality knowledge base may be by can of one so that a system as may from over time if the knowledge base is not or if software include The authors believe that it is to if will work or if the for Healthcare Information is the for such can be demonstrated to be and The authors believe that a framework should be developed and to promote adoption of e-prescribing systems in outline the that such a framework might both now and in the system must improve over time in a and are limited to e-prescribing systems and their and do not address more comprehensive medication safety such as of both patients and medications and of medication orders and doses at the point of care to A approach to what has been and will be through e-prescribing is to making Electronic prescribing cannot an activity is developed for its in settings academic cannot be in alone should not patients e-prescribing system that is the patient information from an system can produce clinical should in The governmental and should the development of standards for transmission, and monitoring of information and play a in the development and of quality drug information knowledge The United States has taken a through in developing standards for the prescribing and dispensing of The and related and Drug products represent excellent A step would be to develop consensus standards for e-prescribing information For example, a is needed for and drug including the and of standards for the and of interactions within drug information should be each vendor its own systems for such information and only this to an of e-prescribing as reported by et in this but it potentially in which, for example, a CPOE system cannot to pharmacy system (or what is to be potentially for a patient the CPOE system might as to but the pharmacy system might not be to the (or A standard for drug information (or for adverse effects of individual should have an it might for example, the following of the that (or of the drug with an adverse a but standard set of for the clinical of the interactions A the of drug A and drug or A of drug an a of the of the base for the based on in or to based on knowledge of but reported in reported as or reported as a of done replicated by multiple a for the of but or potential for or or potentially or and to in a on a of how often each has been reported to for example, such a might 5 more of to of one in to in to such an standard set of for drug adverse effects and drug e-prescribing systems could more determine how to the of alerts to the for and potentially more alerts in a more manner alerts. There are a number of academic and commercial developers who now have from developing e-prescribing systems. such as up” information (such as current patient or with a or when the is to a if the system laboratory results medication ordering (such as and levels when an is it is important to the and time that the laboratory results were a physician might and as being current when ordering today's when in the laboratory results are (and might if the clinician In the current issue of et al. that the same delivered through different interface can to different by their interface to a for drug information The should develop a that on the of et and the general recommendations of et to useful information for future developers and for to install e-prescribing systems. and academic centers in should development of e-prescribing standards as for both information and consider their drug information knowledge to be and and as a result, they are not to an of by In the current environment, e-prescribing systems may not even have access to the drug information knowledge underlying the systems, making quality a In an both their own systems with such and the same systems with the same would produce the same or can and of for drug information knowledge for e-prescribing system software might drug information knowledge to a should that the clinical standard of care in the such as weight-based dosing for is in e-prescribing systems. When it is should be and system under such circumstances might be if weight-based dosing is not the system could not generate prescriptions for should be developed that provide to which vendor and systems have been evaluated at what in time the to the results or academic medical centers should develop based on of vendor (or e-prescribing systems that are available for which of systems or system features (such as the desiderata of the Joint CDS WG) are recommended for in various A approach might be might the current of practice environment on a that with of the would the of the e-prescribing a solo practitioner in an a small a a practice or with a to medical a small a a hospital and a academic medical would various of systems (or system as or and the resources in and information required to use recommendations for e-prescribing systems would with each so that a solo practitioner with e-prescribing system now might be to one of a number of computer-based e-prescribing systems and to only use it for prescribing when needed as to not practice but to provide information and in any By contrast, for a academic medical with an electronic medical record the might of a e-prescribing approach and relevant to use in selecting such a system. organizations, and academic medical centers should promote and for adverse with a system for is an important first the authors that only the has the and resources to the of to a common, of drug information to support e-prescribing over efforts to as as the National Library of to and the Medical can as for how a drug information knowledge base could be The has a among governmental academic medical and commercial to develop and a useful the such an the drug information of medications might include recommended doses for each medication dose form by and and interactions with and of quality of number of of the and other and The drug information knowledge base could be to and commercial In the of a drug information knowledge developers and should both the drug information underlying their e-prescribing products and the related software to for and quality (which will should through and academic centers and to develop the to such and develop standard for the should the results of of their products they are must evolve the and at local to drug information making them more to and to from the should in any governmental to a drug information knowledge base recommended now developing and drug information knowledge and e-prescribing systems not of if the at future provides a drug information knowledge base for all to The could the knowledge base at a and by both quality information resources and useful e-prescribing software with decision support through such as the Health Information and Management Systems should be to develop a system that from and the to provide to their systems on a or or basis to improve patient A set of standards to should be developed for the e-prescribing that e-prescribing systems have the to products and to feedback for quality must determine that the burden of and using e-prescribing systems not the quality of even if the quality of prescribing is For example, if the of using e-prescribing is for each practitioner to patients the net of an excellent system may be individual care providers must first to use e-prescribing systems, must and must provide feedback for quality By patients that the provider is using an e-prescribing the provider the and or use of and the provider alerts the patient in the patient is by a which to be an of a The time for developing a approach to e-prescribing is at The desiderata by the Joint CDS WG present an important step this list of desiderata is it is only a first should be developed over time to for among users, settings, and information systems. In the authors note that individual e-prescribing recommendations may not be for all settings pediatric on of the recommendations are and will to improve the guidelines over is including the of effects of such systems, with development of reliable to in a The authors believe that of such systems will be complex and we the and of that patient safety cannot be done in an manner or to and correct as they work will be required through of governmental academic and The authors applaud the Joint CDS WG recommendations as a step that will evolve based on feedback and in

BACKGROUND: Unintended consequences of computerized patient care system interventions may increase resource use, foster clinical errors, and reduce users' confidence. OBJECTIVE: To evaluate three successive interventions designed to reduce serum magnesium test ordering through a care provider order entry system (CPOE). The second, modeled after a previously successful intervention, caused paradoxical increases in magnesium test ordering rates. DESIGN: A time-series analysis modeled weekly rates of magnesium test ordering, underlying trends, the impact of the three successive interventions, and the impact of potential covariates. The first intervention exhorted users to discontinue unnecessary tests recurring more than 72 hours into the future. The second displayed recent magnesium, calcium, and phosphorus test results, limited testing to one test instance per order, and provided education regarding appropriate indications for testing. The third targeted only magnesium ordering, displayed recent results, limited testing to one instance per order, summarized indications for testing, and required users to select an indication. PARTICIPANTS: Clinicians at Vanderbilt University Hospital, a 609-bed academic inpatient tertiary care facility, from 1998 through 2003. MEASUREMENTS: Weekly rates of new serum magnesium test orders, instances, and results. RESULTS: At baseline, there were 539 magnesium tests ordered per week. This decreased to 380 (p = 0.001) per week after the first intervention, increased to 491 per week (p < 0.001) after the second, and decreased to 276 per week (p < 0.001) after the third. CONCLUSION: A clinical decision support intervention intended to regulate testing increased test order rates as an unintended result of decision support. CPOE implementers must carefully design resource-related interventions and monitor their impact over time.

OBJECTIVE: Authors evaluated whether displaying context sensitive links to infrequently accessed educational materials and patient information via the user interface of an inpatient computerized care provider order entry (CPOE) system would affect access rates to the materials. DESIGN: The CPOE of Vanderbilt University Hospital (VUH) included "baseline" clinical decision support advice for safety and quality. Authors augmented this with seven new primarily educational decision support features. A prospective, randomized, controlled trial compared clinicians' utilization rates for the new materials via two interfaces. Control subjects could access study-related decision support from a menu in the standard CPOE interface. Intervention subjects received active notification when study-related decision support was available through context sensitive, visibly highlighted, selectable hyperlinks. MEASUREMENTS: Rates of opportunities to access and utilization of study-related decision support materials from April 1999 through March 2000 on seven VUH Internal Medicine wards. RESULTS: During 4,466 intervention subject-days, there were 240,504 (53.9/subject-day) opportunities for study-related decision support, while during 3,397 control subject-days, there were 178,235 (52.5/subject-day) opportunities for such decision support, respectively (p = 0.11). Individual intervention subjects accessed the decision support features at least once on 3.8% of subject-days logged on (278 responses); controls accessed it at least once on 0.6% of subject-days (18 responses), with a response rate ratio adjusted for decision support frequency of 9.17 (95% confidence interval 4.6-18, p < 0.0005). On average, intervention subjects accessed study-related decision support materials once every 16 days individually and once every 1.26 days in aggregate. CONCLUSION: Highlighting availability of context-sensitive educational materials and patient information through visible hyperlinks significantly increased utilization rates for study-related decision support when compared to "standard" VUH CPOE methods, although absolute response rates were low.

Peer review is defined as “an evaluation by experts of the quality and pertinence of research or research proposals of other experts in the same field.”1 Peer review is a key component of the process by which an academic journal retains its quality and scientific rigor. Peer review serves three main purposes: providing a quasi-objective metric for the quality of journal submissions, serving as a mechanism to improve the quality of the content, and providing a mechanism for informing and educating journal contributors (authors). Reviewers help editors to determine whether a manuscript is worthy of publication in the specific journal. Not only do reviewers give their opinion of the merits of the manuscript on a “publishability” scale, they explain to the Editorial Office why they ranked the submission as they did and provide suggestions for how to improve the manuscript to both editors and authors. This process allows a discipline to maintain and improve the quality of its published papers. This process secondarily affects what research is conducted, what methodologies are employed, and what messages are disseminated to professional and lay audiences and, to some extent, rightly or wrongly, provides a mechanism to assist in evaluating the work of the authors who submit publications. Peer review is at first glance a thankless job for the anonymous reviewer. It takes time and effort to do well, for which the primary reward is the contribution the reviewer makes to the academic and professional communities served by the journal. However, one gains skill and knowledge from participating in the peer-review process. Reviewers learn effective methods for organizing and presenting scientific content from the authors who submit manuscripts. They also learn from the scientific ideas in the manuscripts they review (even though they cannot act on them or discuss them with others until the manuscript is published in the public domain). Reviewers gain knowledge, understanding, and perspective from reading the confidential reviews (distributed by many journals, including JAMIA, among all reviewers) of other peer reviewers who have submitted opinions regarding the same manuscript. Reviewers also learn from observing the process through which authors iteratively respond to (or fail to respond to) critiques in successive revisions of a manuscript. Despite the longstanding tradition of this approach to manuscript quality control, the medical literature is replete with concerns about the peer-review process. A 1994 survey of authors submitting papers to Journal of Clinical Anesthesia noted that unclear comments, judgmental reviews, discrepant reviews, and disorganized management of the peer review process (including timely return of manuscripts and selection of knowledgeable reviewers) were primary reasons for dissatisfaction.2 These concerns are not limited to specific journals; in fact, prescriptions for constructing an acceptable review have been published for many leading journals.3–5 Given the importance of this task, and the widespread need for peer reviewers in all scientific disciplines, it would seem prudent to incorporate formal training about peer review into the training of academic professionals.6 In fact, some journals, such as the Annals of Emergency Medicine,5 have established training programs for new reviewers. It is of interest, however, that training and guidelines generally are provided after a reviewer has been given peer-review responsibility. Ideally, these practices would be more valuable if provided to new reviewers before they participate in official reviews, rather than through “on-the-job” training. This year, through the combined efforts of its editor, associate editors, and assistant editor, JAMIA designed and implemented a Student Editorial Board (SEB). This board was formed by selecting outstanding trainees who applied from the National Library of Medicine Training Program sites. Each program was invited to submit the names of up to two trainees to the Assistant Editor. Applicants were all well qualified, and had interests and experience spanning the breadth of biomedical informatics. After discussion and voting, the Associate Editors selected six trainees to form the inaugural Student Editorial Board. The current SEB members are: Tricia A. Thornton, BA (Vanderbilt University) Michael F. Chiang, MD (Columbia University College of Physicians and Surgeons) Peter Mork, MS (University of Washington, Seattle) Adam Rothschild, MD (Johns Hopkins University School of Medicine) Roderick Y. Son, MS (University of California, Los Angeles) Lisa J. Trigg, MN, ARNP (University of Washington, Seattle) The SEB is designed to provide hands-on experience to prospective biomedical informatics journal reviewers. We envision that this experience will serve a variety of purposes. First and foremost, the SEB experience will allow prospective reviewers to learn how to construct a well-written, critical, and constructive review. To accomplish this objective, each SEB member is given a monograph written by the JAMIA Editor and Assistant Editor, citing previous work of Stead et al.,7 Friedman,8 and Aydin.9 This monograph describes the elements of a good review, the ethical responsibilities of peer reviewers, and the generally acceptable tone and style of a critical review. Each time an SEB member completes a review, the lessons contained within the monograph are reinforced by a careful critique of each SEB review by the Assistant Editor . In addition, SEB members receive copies of the other JAMIA-assigned reviewers' comments once the editorial process has been completed on the submitted manuscript they reviewed. The SEB experience also will help SEB members understand how to publish work done at any stage of an informatics project. We will ask SEB members to review all types of materials submitted to JAMIA—including books, original research articles, viewpoint documents, model formulation papers, and case reports. We also will provide SEB members with revised and resubmitted manuscripts, so that they might see how a paper evolves from its first reviewed form all the way through to its final draft. Finally, we expect SEB members to learn the time management skills necessary to incorporate ad hoc, but time-critical work into their busy schedules, so that they do not find the process as onerous during their early years as independent investigators. Members will have an opportunity to complete six reviews (provided at times when their own publications or presentations should not compete for their attention). They are expected to adhere to the timeline provided to other reviewers. We will accomplish these objectives by integrating SEB members into the existing JAMIA review process. Each SEB member has provided the JAMIA Assistant Editor his or her areas of interest as a part of the application process. Students are expected to construct a timely, professional review of the manuscript with the same turnaround as regular reviewers. Student reviews are returned to the JAMIA office, where the Assistant Editor reviews them, provides constructive feedback, and accepts revisions by the SEB member. Once approved, the JAMIA Editor includes the student reviews as a part of the overall critique of the manuscript returned to authors. A label informs the author that the review is from an anonymous SEB member. The student reviews are intended to complement the reviews by JAMIA Editorial Board members and outside reviewers that will continue to occur. Students also have an opportunity to turn back manuscripts (not review them) when the timing of a JAMIA request is not convenient for them. Student Editorial Board members—like other editorial board members—are not financially compensated for their work. The major benefits to SEB members are education about the publication process and the construction of a useful review, as well as the notoriety from being selected. Student Editorial Board members also are recognized through a listing in JAMIA on the same page/e-location where the Editorial Board membership is listed. Thus far, the SEB process has proceeded smoothly. Our six SEB members recently completed their first set of reviews. A careful critique of these reviews reinforced the potential impact that the SEB will have on the written critiquing skills of its members. For example, critiques of these reviews were able to use specific textual examples to teach: The fine line between providing constructive insight to the author and clouding insight with condescending statements The importance of including an evaluation of prior work in the introduction of an original research manuscript The impact of poor author writing style on the ability of a reviewer to critique a potentially groundbreaking research project How the author's receptivity toward a review may be affected by the style of the review (particularly when the review addresses the author as “you”) Although it is too early to determine whether this process makes a difference in either the quality of our reviews or the skills of potential reviewers, comments from current SEB members suggest that the process is both enjoyable and instructive. They have been uniformly responsive to critiques of their reviews, and have made revisions that have generally produced a much more effective review. We intend to evaluate the effects of the SEB process more carefully over the year. We will soon expand the SEB from six to 12 members, with a plan to provide each SEB member with at least six reviews during his or her two-year tenure on the Board. The next round of applications will extend beyond trainees in U.S. training programs. In the meantime, we will evolve the SEB to meet the needs of our initial cohort of members, while striving to keep the process both fun and educational.

Care providers' adoption of computer-based health-related documentation ("note capture") tools has been limited, even though such tools have the potential to facilitate information gathering and to promote efficiency of clinical charting. The authors have developed and deployed a computerized note-capture tool that has been made available to end users through a care provider order entry (CPOE) system already in wide use at Vanderbilt. Overall note-capture tool usage between January 1, 1999, and December 31, 2001, increased substantially, both in the number of users and in their frequency of use. This case report is provided as an example of how an existing care provider order entry environment can facilitate clinical end-user adoption of a computer-assisted documentation tool-a concept that may seem counterintuitive to some.

We commend Shiffman and colleagues (“Bridging the Guideline Implementation Gap: A Systematic, Document-Centered Approach to Guideline Implementation”1) for highlighting the challenges of integrating guidelines into clinical practice and proposing pragmatic mechanisms for addressing them. We note, however, that the approach advocated by Shiffman et al., as well as by numerous other groups recently,2–8 is fundamentally a document-centric model. This approach may lead others to assume that representing a guideline correctly as a “computer-readable” document is the majority of the work required for implementation success. Although the “understanding” and representation of the clinical content of a guideline are a sine qua non for its local implementation, the document-centric approach leaves a substantial gap between the idealized document model and any specific guideline implementation in a local clinical system. This considerable gap is not unlike the “curly braces” problem documented for the Arden Syntax a decade ago.3–5 We estimate that 90% of the effort required for successful guideline implementation is (and must be) local, and the remaining 10% of the effort involves “getting the document right.” We believe that an alternative approach to local guideline implementation is to focus on the guideline's recommended actions; on the capabilities of the local care provider order entry (CPOE) or electronic health record (EHR) system that will serve as the “effector mechanism” for the guideline; on locally available computational and clinical resources; and on the guideline's required “clinical infrastructure.” We believe that guidelines should be implemented locally and directly (with a systematic approach, as described below) via local clinical systems (as opposed to a quasi-automatic implementation using a computer-readable, nationally disseminated document). The goal of both the “document-centric” and the “locally customized and guided” approaches is the same: implementation of locally effective guidelines that appropriately influence clinical decision making, resulting in desirable actions that improve patient outcomes. Local guideline implementation requires the following understanding, resources, and efforts: A local, clinically expert champion (or group of champions) who will customize the national guideline to be compatible with local capabilities and practices and, more importantly, take ownership of guideline evolution locally over time. While national-level guidelines should form the basis of evidence-based practice, the unfortunate truth is that national guideline developers rarely reconvene to systematically update guidelines in a timely manner. Unless local experts take responsibility for guideline implementation in the present, and for future updates, the institution where a national guideline is implemented over time becomes at risk of practicing “the ‘perfect’ medicine of bygone eras” (e.g., national standards from 5 to 10 years ago). A locally developed consensus among clinicians across services on how to implement each guideline (e.g., across Medicine, Surgery, Obstetrics/Gynecology, Pediatrics, Emergency Department). This may include modifying the national guideline in various ways: Guideline distillation and presentation. What are the resulting local actions (orders) of the guideline? After focusing on the orders, determine whether the conditional statements qualifying the orders lend themselves to explanatory text or whether a more sophisticated “advisor” program with complex calculations is required. The choices may result in implementation of a simple guideline (e.g., evidence-based “pick-lists” specifying care options for a patient admitted with acute coronary syndrome) as an “order set.”9 Order sets for the most part leave “branching logic” choices up to the end user by providing textual instructions over each subsection of orderables (e.g., “Select one of the following beta-blockers that is most appropriate for the patient from the list below.”). In contrast, another, more complex guideline might require implementation as a programcode–based algorithm (“advisor”) that calculates patient-specific doses and recommendations based on known patient demographics, clinical parameters (e.g., renal function, current orders, clinical diagnoses, weight, height), and laboratory results (e.g., coagulation studies, renal function tests, serological results). For example, Web-based advisors are helpful for ordering multicomponent total parenteral nutrition in a neonatal intensive care unit, which requires careful balancing of electrolytes, fluids, caloric sources, and involves many patient-specific “rules.” Web-based advisors are also helpful for combining complex textual instructions with patient-specific calculations such as for antibiotic selection and dosing based on clinical indications. Note, however, that hardware and software issues locally may influence guideline implementation choices. An old MS-DOS® character-based interface cannot support the bandwidth for complex advisors that a Web-based, multitiered architecture can support, but the former may be at least as good for implementing simple pick-lists. Guideline interpretation/translation locally. How are radiology procedures and the pharmacy formulary represented locally in contrast to the text in the guideline? What orderables inferred in the guideline can actually be ordered in the local clinical system? (e.g., if Doppler studies of the legs are suggested in a guideline for “diagnosis and treatment of suspected deep venous thrombosis,” does the local system allow unilateral or bilateral Doppler ordering and in a manner that is consistent with guideline recommendations?) Do new orderables need to be added or will comments or additional parameters within existing orders suffice to fulfill guideline requirements? Guideline creation based on informed decision making about optimal local methods. Having decided on the appropriate orderables (actions) and whether to create the guideline as an order set or an advisor, what additional details must be specified? If the guideline is best implemented via order sets, should there be a single order set or several linked (nested) order sets (e.g., a main order set and another order set for medications that might already have been created for a different purpose)? If the guideline is more complex, and a multifaceted “advisor” is necessary, can the advisor be simple and one pass (e.g., total parenteral nutrition ordering in a neonatal intensive care unit) or will it involve multistage “component” advisors? For example, for an “anticoagulation advisor,” the initial phase might consist of ordering deep venous thrombosis prophylaxis for “normal” patients at bed rest. The next phase might involve ordering the correct diagnostic procedures, baseline laboratory tests, and “coverage” anticoagulants for patients with “suspected deep venous thrombosis.” The next phase should advise users how to initiate “definitive therapy” per national guidelines once a diagnosis of “deep venous thrombosis” is firmly established. Finally, the last phase might consist of adjusting heparin (or low molecular weight heparin) doses per national guidelines after initial therapy was ordered, based on follow-up information (such as activated partial thromboplastin test results for monitoring heparin dosing). Such multistage protocols must “recognize” the previous state of the patient in the sequence of the protocol, as well as be able to “trigger” the next step on appropriate cues. Often, for such complex guidelines, the code underlying the clinical system may also have to be modified to handle such convoluted, multistage protocols if they are “new and unique” in the experience of the CPOE or EHR system. The approach taken will depend on the resources available, the style of the institution, and the capabilities/flexibilities of the clinical system. A host of reference material supporting guideline implementation. With respect to run-time guideline activation, many reluctant clinicians take a “show-me” stance. In such settings, helpful “educational” links explaining both the rationale for the guideline's suggestions (i.e., the “evidence base” for the guideline) as well as detailed explanations of the procedural steps involved in implementing the guideline, including explicit displays of the calculations/logic performed by the program in a patient-specific manner (e.g., dose calculations) are required. Internet and intranet links to national Web sites, locally maintained “expert” monographs, and documents that describe local hospital policy and procedures must support local guideline implementation. Such references require local maintenance above and beyond any “national” guideline content maintenance per se. An organized set of ancillary “EHR system-based” information relevant to clinicians' thought processes to support their attempts to follow a guideline. For example, if the information related to the guideline requires concurrent awareness of active orders, medication doses, laboratory results, etc. (as might occur for a heparin therapy advisor), how will the information be obtained and displayed in “real time” using the underlying CPOE or EHR system as part of the “guideline display page” so that the clinician has “one stop shopping” for guideline-related decision making? There are no pragmatics at the national level for how to do this in individual local systems. A method for advertising the availability/applicability of the guideline for appropriate user groups. This might consist of Placing a guideline order set (if it exists in this form) in the default list of selectable order sets for nursing units on which the guideline is likely to be applicable. Automatically triggering the order set or advisor based on when the user enters specific orders or when specific real-time laboratory results occur. This ability is dependent on the flexibility of the clinical system. A set of parameters and methods for tracking and measuring guideline effectiveness. If the CPOE (local guideline implementation) system does not distinguish in its “log files” (system database) which orders were entered “free hand,” which orders were entered via which “order set,” and which orders were created using a specific “advisor program,” then determining the situations in which guideline suggestions were being followed may become difficult or impossible. Guideline implementers must be able to determine both when a user was prompted to follow a guideline and whether the user chose to do so (and optimally to record why a guideline suggestion was not followed through user-generated explanatory text). The mechanics of doing so are dependent almost wholly on the local system and cannot be specified as part of a nationally distributed “reference guideline document” that is quasi-automatically incorporated into a system (although this might be possible for groups of users using the same clinical system). In addition, because a published guideline is a snapshot in time, local changes in practice and expertise may lead to subsequent customization of some guidelines, so a mechanism for “versioning” both guidelines and their effecter order sets and advisors is required in the local clinical system. The above-described tasks highlight the guideline disseminators' and guideline implementers' mutual problem: Although there is a desire to have a “top down,” document-centric representation that fully describes each guideline, such representations cannot offer pragmatic, easily assimilated, maintainable, and actionable mechanisms for guideline incorporation into local production systems, nor can they do so in a manner that effectively integrates the guideline into local workflows. Pryor and Hripcsak's10 sharing of “rather simple” Arden syntax medical logic modules (MLM) between two institutions a decade ago began to reveal the magnitude of effort required for local integration. In their example, seven MLMs required 43 modifications to be translated between two systems that had already “adopted” the ASTM-standard MLM. A decade later, their conclusion is still applicable: “Standards can be of great assistance in sharing the work of many, but the routine sharing of medical knowledge may be delayed until common standards exist not only in the description of the logic but in all aspects of the medical information system.” As Bates11 suggests, simplicity in implementation often is most effective. Clinical end users sometimes resent overly complex, multiple-screen advisors even when they convey best practices. In implementing clinical systems, it is important to remember that the clinician-user is both more intelligent and more knowledgeable and understanding of the patient's condition than is the clinical computer system. If one relies on the intelligence of the end user (clinician) as a component of guideline implementation and execution, very simple guideline representations, such as order sets, may suffice. Additionally, clinicians should have the ultimate discretion in patient care, including guideline implementation. There are more exceptions than rules in clinical practice. As long as clinicians are made aware of relevant guidelines “just in time” during patient care activities, “optimal” guideline compliance rates may be 80% and not 100% due to patient-specific factors not considered by guideline developers. Effective guideline implementation is hopelessly intertwined with considerations based on local clinical applications, local clinical practices, and local control of procedures and policies as they evolve over time. National document-centric representations of guidelines, although helpful and important, must be seen as supporting the local pragmatics of implementing guidelines on the front lines, and increasing emphasis should be placed on the latter. The situation is not hopeless in that the work that local institutions must do to adopt and maintain guidelines can sometimes be shared “locally” among hospitals and clinics belonging to a conglomerate “system” that share a common information system infrastructure or “locally” among the “user group” of a national vendor with multiple install sites, all of whom presumably share the same implementation platform. The best methods for evolving and supporting guidelines, once they are “installed” will remain an active area of both research and practical interest as EHR and CPOE systems become more ubiquitous.

William W. Stead, MD, recently completed a decade of service to AMIA and to our profession as the founding editor of the Journal of the American Medical Informatics Association. We, the associate editor group who worked under him, take this opportunity to thank him for his profound contributions. Dr. Stead imparted a number of important characteristics to the editorship and to JAMIA by demonstrating: A massive, passionate commitment to moving the field forward through dissemination of scholarship and innovation. A commitment to scientific rigor and quality of publication, sometimes at the expense of decreased page count. A strong desire to represent the breadth and depth of this multidisciplinary field. A long view on the course of events, favoring broad perspectives over provincial ones. A palpable, deep and genuine respect extended to every author and reviewer, often expressed through tactful, direct, and honest feedback. An ability to find the kernel of good and truth in almost any paper and to help authors find it also—sometimes by working with authors to rewrite manuscripts interactively. A polite but persistent approach to editorial disagreements, combined with a polite and good natured acceptance of editorial advice, even when it differed from his perspective. A ready opinion—usually strongly held, on any topic—balanced by a willingness to remodel himself and his skill set to adapt to changing times. Bill Stead has been a visionary leader in biomedical informatics. He has brought extraordinary perspective and insight into creating a new journal “from scratch.” In an era when academic “triple threats” are said to be nearing extinction, he has shown that one can still understand and support clinical practice, conduct research and build an academic enterprise and contribute to excellent institutional management. One of his favorite expressions is, “We shouldn't do it if it isn't scalable and sustainable,” which he has applied both to development of informatics systems at Vanderbilt, and to practices within JAMIA. His keen business sense has helped to make JAMIA a break-even endeavor for the American Medical Informatics Association from its inception. His experience in team building has led to a cohesive and coordinated editorial board and staff that turns around submissions with exemplary efficiency. And on top of all that, he has big shoes. In recognition of his pioneering leadership, the American Medical Informatics Association will initiate a Janet and Bill Stead Lectureship (details forthcoming later). Thank you, Dr. Stead, for creating the “industry standard” for excellence in medical informatics editorial leadership.

OBJECTIVE: To describe the development and evaluation of computational tools to identify concepts within medical curricular documents, using information derived from the National Library of Medicine's Unified Medical Language System (UMLS). The long-term goal of the KnowledgeMap (KM) project is to provide faculty and students with an improved ability to develop, review, and integrate components of the medical school curriculum. DESIGN: The KM concept identifier uses lexical resources partially derived from the UMLS (SPECIALIST lexicon and Metathesaurus), heuristic language processing techniques, and an empirical scoring algorithm. KM differentiates among potentially matching Metathesaurus concepts within a source document. The authors manually identified important "gold standard" biomedical concepts within selected medical school full-content lecture documents and used these documents to compare KM concept recognition with that of a known state-of-the-art "standard"-the National Library of Medicine's MetaMap program. MEASUREMENTS: The number of "gold standard" concepts in each lecture document identified by either KM or MetaMap, and the cause of each failure or relative success in a random subset of documents. RESULTS: For 4,281 "gold standard" concepts, MetaMap matched 78% and KM 82%. Precision for "gold standard" concepts was 85% for MetaMap and 89% for KM. The heuristics of KM accurately matched acronyms, concepts underspecified in the document, and ambiguous matches. The most frequent cause of matching failures was absence of target concepts from the UMLS Metathesaurus. CONCLUSION: The prototypic KM system provided an encouraging rate of concept extraction for representative medical curricular texts. Future versions of KM should be evaluated for their ability to allow administrators, lecturers, and students to navigate through the medical curriculum to locate redundancies, find interrelated information, and identify omissions. In addition, the ability of KM to meet specific, personal information needs should be assessed.

Joan S. Ash Joan S. Ash Joan Ash is Associate Professor, Division of Medical Informatics and Outcomes Research, School of Medicine, and Associate Professor, Libraries, Oregon Health & Science University (OHSU), Portland, OR. She received her B.A. degree from Emmanuel College and holds masters degrees in library science, health science, and business administration, from Columbia University, California State University, Northridge, and Portland State University, respectively. Her doctorate is in Systems Science, Business Administration from Portland State. Dr. Ash's prior positions include IAIMS Coordinator at OHSU and Associate Director of OHSU Libraries. She has also served as the Assistant Director at the University of Connecticut Health Center Library and as Senior Reference Librarian at the Yale Medical Library. In addition, Dr. Ash has worked as Director of the OHSU Oral History Project and as an NLM Fellow in Applied Medical Informatics. Dr. Ash's research focus is organizational behavior as it relates to health information system implementation. Her work studying and evaluating the implementation of computerized physician order entry systems and order communication, applying diffusion of innovations theory related to health information systems, and utilizing qualitative methods in informatics has highlighted the importance of the behavioral aspects of systems implementation. She authored the paper that received the first annual Diana Forsythe Award (for work at the intersection of the social sciences and informatics) from the American Medical Informatics Association. Her publications have appeared in journals pertaining to public health, academic libraries, medical libraries, management, and informatics. She has authored a book on health information resources. Professionally, Dr. Ash has been active in the Medical Library Association and the American Medical Informatics Association. She was elected to the board of directors and selected for the journal editorial boards of both. She has served as a member of the Biomedical Library Review Committee of the National Library of Medicine and special emphasis panels for the NIH National Center for Research Resources. She has now been elected to membership in the American College of Medical Informatics. Philip Bourne Philip Bourne Philip Bourne is a Professor in the Department of Pharmacology at the University of California, San Diego, the Director of Integrative Biosciences at the San Diego Supercomputer Center and Co-director of the Protein Data Bank. He received a first class honors degree and a PhD in Chemistry from the Flinders University of South Australia. Dr. Bourne was a postdoctoral research fellow at both Sheffield University, UK, and Columbia University, NY, where he worked on the elucidation of various protein structures including the iron storage protein ferritin and post-synaptic neurotoxins. He began a career in structural bioinformatics nine years ago and focuses on data modeling, query languages and the understanding of structure-function relationships with particular reference to cell signaling and apoptosis. Dr. Bourne is an Associate Editor of the journal Bioinformatics and serves on the Advisory Board of Biopolymers. He is past-president of the International Society for Computational Biology and the recipient of Sun's convergence award. He has now been elected to membership in the American College of Medical Informatics. Simon P. Cohn Simon P. Cohn Simon P. Cohn, MD, MPH, is the National Director for Health Information Policy for Kaiser Permanente. Dr. Cohn earned both his BA and MPH degrees at the University of California, Berkeley and his medical degree from the University of California, Davis. He completed his postgraduate medical training at St. Mary's Hospital in San Francisco. He is board certified in Emergency Medicine and a Fellow of the American College of Emergency Medicine. Dr. Cohn maintains an active clinical practice. Dr. Cohn's medical informatics interests began in 1984, when he developed widely used computer-assisted medical charting applications using the Apple Macintosh computer for ambulatory and Emergency Department settings in Northern California Kaiser Permanente. He holds two copyrights for these software applications. In 1991 he was named Clinical Information System Coordinator for Kaiser Permanente. In this role he worked with other KP leaders to develop and implement the first national KP Clinical Information Systems Strategy. From 1997–98 he was the National Director for Data Warehousing for KP. Since 1999, he has devoted his energies at KP to health information policy and issues related to encounter data capture. Dr. Cohn was co-investigator on a federally sponsored demonstration project for clinical terminologies. He has published on this topic. In 2002, he was a recipient of a President's Award from AMIA for his contributions to the field. Dr. Cohn serves on many national boards and committees concerned with health information policy issues. He is a member of the National Committee on Vital and Health Statistics (NCVHS), the main public advisory committee to the U.S. Department of Health and Human Services on health information policy and HIPAA. He chairs its Subcommittee on Standards and Security. Under his chairmanship, the subcommittee, in addition to its central role in HIPAA implementation, has developed a series of national recommendations on standards for patient medical record information. Additionally, he is a member of the AMA CPT Editorial Panel and the National Uniform Claims Committee (NUCC). He is also a member of the Institute of Medicine's Committee on Patient Safety Data Standards. Previously Dr. Cohn served as a board member of the Workgroup on Electronic Data Interchange (WEDI) and Executive Committee member of the Computer-based Patient Record Institute (CPRI). He has now been elected to membership in the American College of Medical Informatics. Antoine Geissbuhler Antoine Geissbuhler Antoine Geissbuhler is a Professor of Medical Informatics at Geneva University School of Medicine and Director of the Division of the Medical Informatics at Geneva University Hospitals. Dr. Geissbuhler graduated from the Geneva University School of Medicine in 1991 as a Philips European Young Scientist first award laureate. He received his doctorate for work on tri-dimensional reconstruction of positron emission tomography images. He then trained in internal medicine under the direction of Prof. Francis Waldvogel. After completing a post-doctoral fellowship in medical informatics at Vanderbilt University, he rose to Associate Professor of Biomedical Informatics and Vice-Chairman of the Division of Biomedical Informatics at Vanderbilt University Medical Center. There Dr. Geissbuhler worked primarily on the development of clinical information systems and knowledge-management tools. During his time at Vanderbilt, he was the primary developer of the WizOrder computerized Physician Order Entry system, which uses integrated decision support and is now being marketed commercially. In 1999, Dr. Geissbuhler returned to Geneva to head the Division of Medical Informatics in Geneva University Hospitals and School of Medicine, following in the steps of Prof. Jean-Raoul Scherrer, who founded this world-renowned group. Dr. Geissbuhler's current research focuses on the development of innovative computer-based tools for improving the quality and efficiency of care processes, at the local level of the hospital, at the regional level of a community healthcare informatics network, and at the global level with the development of a telemedicine network in Western Africa. He has now been named an international associate of the American College of Medical Informatics. Dario A. Giuse Dario A. Giuse Dario A. Giuse is Associate Professor of Biomedical Informatics and Associate Director of the Informatics Center, Vanderbilt University Medical Center; and Associate Professor of Computer Science in the Vanderbilt University School of Engineering. Dr. Giuse received the Dr. Ing. (“Dottore in Ingegneria”) degree from the Politecnico di Milano, Milan, Italy in 1979. He also received an M.S. degree (Pure and Applied Mathematical Logic) from Carnegie Mellon University in 1993. After joining the faculty of the School of Medicine at Vanderbilt University, he spearheaded the effort to implement the StarChart electronic patient record system. The system provides an integrated, longitudinal patient record that contains all lab results, radiology reports, discharge summaries, problem lists, clinic notes, letters, echocardiography, cardiac catheterization, and pulmonary function test reports for both inpatient and outpatient encounters. The system is used interactively via a Web-based front-end for day-to-day clinical patient care, and as the back-end repository for automated decision support tools. Before coming to Vanderbilt in 1994, Dr. Giuse was an adjunct faculty member at the University of Pittsburgh Medical Center. There, he served as the main architect of the QMR-KAT knowledge acquisition tool, the first knowledge editor to be used for large-scale, multi-center diagnostic medical knowledge acquisition. With his colleagues at the University of Pittsburgh, he conducted a systematic evaluation of the costs of long-term maintenance of medical knowledge bases, documenting statistically significant inter-rater reproducibility for the extraction of medical knowledge from the literature. Dr. Giuse became a member of Sigma Xi in 1984. He serves on the editorial boards for Artificial Intelligence in Medicine and Computer Methods and Programs in Biomedicine, Elsevier and was selected as a guest editor for a section of the Yearbook of Medical Informatics, Schattauer in 1999. Dr. Giuse has served as a member of the American Medical Informatics Association Bylaws Committee, as well as co-chair of Working Group 10 (Hospital Information Systems) of the International Medical Informatics Association. In 1991, Dr. Giuse was Keynote Speaker at the VI National Congress of the Italian Medical Informatics Association and more recently addressed the International Medical Informatics Association Working Conference on Health Information Systems in 2002. He has now been elected to membership in the American College of Medical Informatics. Nunzia B. Giuse Nunzia B. Giuse Nunzia B. Giuse is Associate Professor of Biomedical Informatics and Director of the Eskind Biomedical Library at Vanderbilt University Medical Center. She received her MLS from the University of Pittsburgh School of Library and Information Science after earning an MD at Universita' degli Studi di Brescia, Brescia, Italy. Prior to coming to Vanderbilt in 1994, Dr. Giuse established herself at the University of Pittsburgh as an independent researcher in the area of multi-center medical knowledge base acquisition strategies. She conducted a systematic sequence of investigations of medical knowledge base methodologies from 1988–95 that represented a significant contribution to the relatively new field of medical informatics. In her current position, Dr. Giuse has pioneered the application of models from the adult learning research literature to continuing professional education within the library. She has also pioneered the idea of actively involving the library in VUMC clinical activities, and has gained national recognition for developing the “Informationist” model. Dr. Giuse currently chairs the Medical Informatics Section, Medical Library Association and has chaired the Medical Informatics Section/MLA Career Develop-ment Grant Jury. She is currently a member of the Journal of the American Medical Informatics Association Editorial Board. In addition, she serves as a member of the Board of Scientific Counselors, Lister Hill Center, National Library of Medicine. A complete list of Dr. Giuse's committee memberships can be found at . She has now been elected to membership in the American College of Medical Informatics. John Lumpkin John Lumpkin Dr. John R. Lumpkin was appointed director of the Illinois Department of Public Health in January 1991, after serving as acting director since September 1990. He is the first African-American to hold this position at the agency. For the previous five years, Dr. Lumpkin had been associate director of the Department's Office of Health Care Regulation. Before joining the state health department, Dr. Lumpkin served as an emergency physician at several Chicago hospitals, including St. Mary's Nazareth, South Chicago Community and the University of Chicago Hospitals and Clinics. Dr. Lumpkin received his medical degree in 1974 from Northwestern University Medical School. He trained in Emergency Medicine at the University of Chicago and earned his master's degree in public health from the University of Illinois at Chicago, School of Public Health. Dr. Lumpkin is active in national policy development on public health information systems and performance measurement and teaches these subjects at the graduate level at the University of Illinois at Chicago, School of Public Health. He has also been active in injury prevention and has provided technical assistance to the Ministry of Health of the Arab Republic of Egypt on behalf of the U.S. Public Health Service. He has served on a number of national advisory committees and currently serves as Chair of the National Committee on Vital and Health Statistics (NCVHS), Chair of the NCVHS Workgroup on National Health Information; past member of the Centers for Disease Control and Prevention's Advisory Committee to the Director; the Institute of Medicine's Committee on Public Health Performance Measures, Public Health Roundtable and Performance Partnership Grants Panel. Active in numerous professional organizations, Dr. Lumpkin is a member of the National Forum for Health Care Quality Measurement and Reporting (i.e., National Quality Forum), past president of the Association of State and Territorial Health Officials (ASTHO), a former member of the Board of Trustees of the Foundation for Accountability, a former Commissioner of the Pew Commission on Environmen-tal Health, a past board member of the American College of Emergency Physicians and past president of the Society of Teachers of Emergency Medicine. He has now been elected to membership in the American College of Medical Informatics. Julie McGowan Julie McGowan Julie J. McGowan, PhD, is Associate Dean for Information Resources and Educational Technology at Indiana University School of Medicine. She holds academic appointments as Professor of Knowledge Informatics and Professor of Pediatrics and is an Affiliated Scientist at the Regenstrief Institute. She received an MLS from the University of Maryland and an MA (Medical Iconography) and a PhD (Medical Education) from the University of South Carolina. Dr. McGowan's primary interest is application of information technology in support of clinical decision-making and medical education. Formerly the Associate Dean for Health Sciences Informatics and Library Resources at the University of Vermont College of Medicine, she led development of VTMEDNET, a comprehensive statewide health information network that provided secure access to patient information, library and public health resources, as well as distributed learning. She is involved in similar citywide projects in Indianapolis through the Regenstrief Institute. Dr. McGowan's interest in informatics education at the undergraduate medical school level led to the development of an innovative four-year Vertical Curriculum in Information Literacy and Applied Medical Informatics at the University of Vermont and the re-engineering of a required fourth year clerkship in Medical Informatics at Indiana University. She is a founding faculty member of the IU School of Informatics, the first in the country, and a member of the participating faculty of the Regenstrief Institute Medical Informatics Fellowship program. Dr. McGowan worked to establish the Group on Information Resources for the Association of American Medical Colleges (AAMC), was a member of its first elected Steering Committee and served as Chair for 2002–2003. She is a member of AMIA's Public Policy Committee, Meetings Committee, and Finance Committee and just completed a three-year term on the Board of Directors of the Medical Library Association [1999–2002]. Dr. McGowan also is a member of the Biomedical Library and Informatics Review Committee of the National Library of Medicine [1999–2003] and the Health Research Dissemination and Implementation Study Section/Special Emphasis Panel of the Agency for Healthcare Research and Quality [1998–2003]. She has now been elected to membership in the American College of Medical Informatics. Lucila Ohno–Machado Lucila Ohno–Machado Lucila Ohno-Machado is Associate Director of the Department of Radiology Decision Systems Group at Brigham & Women's Hospital and Associate Professor of Radiology at Harvard Medical School. She also is an Affiliated Faculty member in the Health Sciences and Technology Division for Harvard and MIT. Dr. Machado earned an MD from the University of Sao Paulo School of Medicine in Brazil, an MHA from Escola de Administracao de Sao Paulo, FGV, Brazil, and a PhD from Stanford University. From 1990 to 1991, Dr. Machado served as the director of the Medical Informatics program in the University of Sao Paulo Radiology department. She has been an Information Technology Services consultant for Kaiser Permanente in Walnut Creek, California and has taught as both an Instructor and an Assistant Professor in the Harvard Medical School Radiology department. Dr. Machado has received numerous honors for her work in medical informatics. She received the Doctoral Dissertation Award from the Agency for Health Care and Policy Research in addition to an award for Best Theoretical Paper in the Student Paper Competition at the Eighteenth Symposium on Computer Applications in Medical Care, American Medical Informatics Association (AMIA). She has also been recognized by AMIA as a Best Paper Award finalist at the 1996 Fall Symposium and as a Martin Epstein award recipient in 1994. Dr. Machado also received the Dean's fellowship award from the Stanford University School of Medicine, the James A. Shannon Director's Award from the National Institute of Health, the Award for Outstanding Contributions to Research from Brigham & Women's Hospital, and the Taplin Award from the Division of Health Sciences and Technology, conferred by Harvard and MIT. She has also given of her time by sitting on several review committees and panels, including the Biomedical Library Review Committee for the National Library of Medicine and the Special emphasis panels of the NIH's National Center for Research Resources, the National Institute of Environmental Health Sciences, and the National Heart, Lung, and Blood Institute. Dr. Machado currently serves as a Reviewer for the Norges Forskningsrad (Norwegian Research Council) and was an Editorial Board member of the Journal of Biomedical Informatics. She has now been elected to membership in the American College of Medical Informatics. Frank Sonnenberg Frank Sonnenberg Frank Sonnenberg, MD, FACP, is Professor of Medicine at the Robert Wood Johnson Medical School of the University of Medicine and Dentistry of New Jersey (UMDNJ) and Clinical Associate Professor of Health Informatics at UMDNJ's School of Health Related Professions. He is Medical Director of Clinical Information Systems of the Robert Wood Johnson University Medical Group, the faculty practice of the Robert Wood Johnson Medical School and Director of Health Informatics at UMDNJ. Dr. Sonnenberg received his BS in Biochemistry with Highest Honors from the State University of New York at Stony Brook and his MD from UCLA. He did a residency in Internal Medicine at UCLA and fellowship in Clinical Decision Making and Medical Applications of Computer Science at the New England Medical Center. Dr. Sonnenberg's research has focused on applications of decision modeling to health care including cost-effectiveness analysis. He has been one of the principal developers of Decision Maker, one of the earliest, and still widely used, microcomputer-based decision analysis programs and U-Maker, a microcomputer-based utility assessment program. Dr. Sonnenberg's current research focuses on user-friendly interfaces to decision models and computer-based implementation of clinical guidelines including HGML, the Hypertext Guideline Markup Language, an XML-based markup approach to clinical guideline encoding. Dr. Sonnenberg is currently Editor-in-chief of the journal Medical Decision Making. He has served on the editorial board of the Journal of the American Medical Informatics Association. He is a previous recipient of first prize in the Lee Lusted Student Prize Competition of the Society for Medical Decision Making and has received FIRST and RCDA Awards from the National Library of Medicine. He has now been elected to membership in the American College of Medical Informatics.

The Health Insurance Portability and Accountability Act of 1996 (HIPAA) carried with it the expectation that compliance would be complete by April 14, 2003. The act includes provisions for Privacy and Security of personal health information as well as for electronic standards for communicating claims data and unique identifiers for health care providers and organizations. Most pertinent to JAMIA are the regulations regarding transmittal of personal patient information. It is important to note that these guidelines apply principally to Protected Healthcare Information (PHI), defined as “a subset of individually identifiable health information (IIHI) that is maintained or transmitted in any form … and relates to the past, present, or future physical or mental condition of an individual; provision of health care to an individual, or payment for that health care; and identifies or could be used to identify the individual.”1 While protecting patient privacy is important, it is also imperative that health care researchers be able to present data to support claims. In order to comply with HIPAA, JAMIA requires that all authors “de-identify” patient information in their text, tables, and figures, by deleting (or replacing with permuted data) the following from any part of the manuscript that contains patient information: Names All geographic subdivisions smaller than a state, including street address, city, county, precinct, zip code, and their equivalent geocodes, except for the initial three digits of a zip code if, according to the current publicly available data from the Bureau of the Census: (1) The geographic unit formed by combining all zip codes with the same three initial digits contains more than 20,000 people; and (2) The initial three digits of a zip code for all such geographic units containing 20,000 or fewer people is changed to 000. All elements of dates (except year) for dates directly related to an individual, including birth date, dates of admission, discharge, tests or procedures, date of death; and all ages over 89 and all elements of dates (including year) indicative of such age, except that such ages and elements be aggregated into a single category of 90 or older. Telephone and/or fax numbers Electronic mail addresses Social security numbers and/or medical record numbers Health plan beneficiary numbers Account numbers Certificate/license numbers Vehicle identifiers and serial numbers, including license plates Device identifiers and serial numbers Patient-related Web Universal Resource Locators (URLs) Internet Protocol (IP) address numbers Biometric identifiers, including finger and voiceprints Full face photographic images and any comparable images Any other unique identifying number, characteristic, or code In addition, JAMIA authors must alter any identifying patient information, such as diagnostic indicators present in a patient problem list, when the combination of attributes might uniquely identify an individual.

The paper in this issue by Hripcsak and Wilcox, “Reference Standards, Judges, and Comparison Subjects: Roles for Experts in Evaluating System Performance,”1 is well written and presents a thoughtful analysis of the topic. As the authors acknowledge, however, there is more to the evaluation of clinical informatics systems than can be accomplished through comparison to experts.2,3 Hripcsak and Wilcox focus on “how to use experts in evaluating systems when one needs them,” whereas this commentary focuses on the question, “when should one use experts as part of a system's evaluation” The two perspectives are complementary rather than contradictory. As noted previously,4 System evaluation in biomedical informatics should take place as an ongoing, strategically planned process, not as a single event or small number of episodes. Complex software systems and accepted medical practices both evolve rapidly, so evaluators and readers of evaluations face moving targets. … [C]urrent thinking recognizes that such systems are of value only when they help users to solve users' problems. Users, not systems, characterize and solve clinical diagnostic problems. The ultimate unit of evaluation should be whether the user plus the system is better than the unaided user with respect to a specified task or problem.… If the ultimate evaluation of a system depends on whether users of the system perform a specified task better when they use the system than when they don't, then there must be public, objective criteria (a “gold standard”) made available before an evaluation begins, to determine the quality of performance of an individual on a task (independent of whether the individual uses a decision-support tool). Hripcsak and Wilcox state that experts can be used in three evaluation settings: to generate, through introspection and expertise, the reference standard per se (e.g., by providing a list of “correct” diagnoses or of “correct” therapeutic interventions based on a reading of the problem at hand); to judge (and label) the individual behaviors of subjects in the study—on a scale ranging from “optimal” through “acceptable” to “inadequate”—without providing an absolute list of “correct answers”; and as actual subjects in the study, to make it possible to rate how well the system performs in comparison with the performance of human experts. Hripcsak and Wilcox's first two scenarios assume that no absolute, independent gold standard is available, so that the experts' opinions represent the next best metric; in their third scenario, a gold standard must exist against which both experts and study subjects are graded in performance. In each of these three settings, as in any formal, summative evaluation of a clinical informatics system, it is best to compare subjects' performances with and without the system, no matter what absolute metric of performance is used. Hripcsak and Wilcox state, Experience shows that accurate reference standards rarely exist; if it were easy to obtain the correct response, a medical informatics system would be unnecessary. However, while it is indeed difficult to find clinical settings in which absolute answers are available, it should be the goal of system evaluators to first ask the question, “Can we design an evaluation for the system that involves use of a reliable, objective, external gold standard?” For example, each clinicopathologic conference published in “Case Records of the Massachusetts General Hospital” in the New England Journal of Medicine involves a definitive procedure (laboratory test, biopsy, or autopsy) as a “gold standard” to establish the patient's correct diagnosis. Some formative system evaluations of diagnostic systems in clinical informatics have used such retrospective published cases to evaluate systems because they provide an external “gold standard.” By definition, such retrospective studies cannot test system performance “on the front lines” of clinical care provision. In situations in which a prospective, summative evaluation of a clinical diagnostic system is required, the evaluation should ideally be performed “on the front lines” at a time when assistance is truly required and no definitive answer is available. Rather than using experts as a gold standard, however, it may often be possible to develop a protocol by which patients can be used as subjects when their diagnoses are unknown, but then patients are closely followed by protocol for a long time until a diagnosis is established by objective, predefined criteria.5 If at the end of the follow-up interval, no diagnosis can be determined by the preset criteria, the case should be labeled as “unable to establish/confirm a diagnosis” and dropped from inclusion in the study. Only when no reliable, external gold standard can be identified should experts' opinions be used. In the area of systems for therapy and prognosis, expert opinions may play a role when randomized controlled studies cannot be carried out. Each patient can only follow their own trajectory of responses to interventions, so if subjects are allowed to select from a set of potential interventions, even if “real” patient case data are used, one can only hypothesize that with a different intervention than was actually used in the case, the patient's outcome might have been different. Ideally, only randomized controlled trials matching patients in the intervention group to patients in the control group, with the objective of tracking and comparing specific outcomes, can determine whether clinicians using decision support tools provide “better” care than the same (or similar) clinicians without decision support tools. Such studies are difficult at best, requiring large numbers of clinicians and patients and long follow-up intervals. Matching physicians with like abilities in control and intervention groups is arduous; matching patients with “equivalent” degrees of “equivalent” illnesses between intervention and control groups is extremely difficult. The use of experts can be misleading in the absence of a gold standard. Imagine a scenario in which patient case records are presented to students who are asked to provide diagnoses, and experts' opinions are sought for use as “gold standard” diagnoses. This may be appropriate if the evaluation of the students' diagnoses is aimed at probing their reasoning abilities. However, consider a different situation, in which instead of actual patient data, a computer-based diagnostic knowledge base is used to generate sample “patient cases.” If a case with findings of “fever, arthralgias, skin rash, and abdominal pain” is presented, would one accept the disease template used to generate the findings, systemic lupus erythematosus, as being the “correct” diagnosis? What if an expert panel determined that with the same nonspecific findings, Lyme disease were the “best” diagnosis? In the absence of a pathognomonic weight of evidence, a “definitive” opinion by experts must be taken with at least one grain of salt, since a truly expert opinion would be that the weight of the evidence in the case could not lead to the conclusion of any specific diagnosis. Experts rarely offer such opinions when they are being consulted as experts. In the current era of evidence-based medicine, the opinions of experts should be tempered by an attempt to measure the “weight of the evidence” that the experts interpret. Even human experts are susceptible to the “garbage in, garbage out” phenomenon.

The 2001 debate of the American College of Medical Informatics focused on the proposition that national regulatory mandate of computer-based provider order entry (CPOE), to take effect by the end of 2005, portends greater benefit than risk for health care delivery. Both sides accepted that provider order entry offers potential benefit. Those supporting the proposition emphasized public safety, noting that payers have little economic incentive to pay for quality and that a mandate would force vendors to improve the usability and value of their systems. They argued that the mandate would align the economic incentives to finally allow CPOE to be widely adopted. Those opposing the proposition emphasized the risks resulting from a mandate, including the direct implementation costs, the logistic issues of implementation, and the cost of failed implementations. They also noted the potential for errors introduced by the systems themselves and the fact that the safety and utility of commercially available CPOE products have yet to be proved.

The vision of integrating information-from a variety of sources, into the way people work, to improve decisions and process-is one of the cornerstones of biomedical informatics. Thoughts on how this vision might be realized have evolved as improvements in information and communication technologies, together with discoveries in biomedical informatics, and have changed the art of the possible. This review identified three distinct generations of "integration" projects. First-generation projects create a database and use it for multiple purposes. Second-generation projects integrate by bringing information from various sources together through enterprise information architecture. Third-generation projects inter-relate disparate but accessible information sources to provide the appearance of integration. The review suggests that the ideas developed in the earlier generations have not been supplanted by ideas from subsequent generations. Instead, the ideas represent a continuum of progress along the three dimensions of workflow, structure, and extraction.

This is the edited transcript of a discussion, among the authors and audience, that followed the presentation that led to the paper “Integration and Beyond: Linking Information from Disparate Sources and into Workflow,” which appears on p. 135. Mark Musen: Bill, when you presented the three generations of integration, the implication was that the third generation is at hand. It was all present tense. I think all of us agree that architectures that allow us to encapsulate knowledge and data in ways that permit reuse are quite exciting. But are we really in the present tense? Have we really achieved these kinds of architectures and, in particular, when you go to the vendor demonstrations, what do you see of this? Bill Stead: That is a very interesting question, because I think the third generation is more in hand than the second. I think “generation” may be the wrong word, because it suggests that the third supplants the second. Instead, techniques from each of the generations coexist in equilibrium. For example, the UMLS provides us with mapping between codes for the kind of things you order for a patient (diagnosis, tests, medications) and the literature. At Vanderbilt we use this mapping to let you ask, “What are the references relevant to the things that have been ordered?” So, to that degree, third generation exists. It is the second generation that is really hard, because it requires regularization. There's a difference between the Vanderbilt vegetable and the Columbia MED, in that the Columbia MED relates the source vocabularies of the various feed systems (third generation), whereas the Vanderbilt vegetable tries to build an enterprise-wide source vocabulary that is then reflected back into the source systems, prealigning their vocabularies. Back to your question, 3M is an example of a vendor that has pursued an architectural strategy. Member of the audience: I think one of the toughest things we all have to deal with is updating our dictionaries. In the simplest cases, the name of an organism is changed and we just have to do the maintenance. It is tougher, when, as with Citrobacter, they do genetic studies and say, “Oh, it's really six different organisms, not one.” We have the human genome project coming very quickly. Even that is just the tip of the iceberg. We're not only going to see all the genes; we're then going to see clinical tests based on gene expression. Essentially, you'll be able to look at something on the order of 180,000 gene products and whether they're up or down regulated. How are we going to integrate such an incredible amount of data at a time when we're going to also be changing how we think about these processes? Classification and simple mapping are not going to work, because the lumpers and splitters are going to be arguing furiously on a daily basis. Randy Miller: The problems you mentioned are clearly on the horizon and very important. But at a simple level, people are people and all of what you're talking about doesn't change how people will present to their primary care providers. At least that part of what exists will not get torn apart. I think what you're talking about is very rich, very vast information overlays on top of what we already have. We don't have to throw out what we have, we need to be ready to extend the linkages. How that will be done is an unanswered question that will result in multiple research grants. Bill Hersh: I think you allude to one of the key points, which is structuring the metadata with the right levels of granularity. Clearly, when we find an organism that can't fit in the existing framework, then that's problem. But if we find that an organism just represents a subcategory of others, and if there's a good hierarchic structure, it can be fit in. The same goes, for example, for diabetes. People classify diabetes with this complication and that complication, but often we just want to know whether the patient has diabetes. Again, a good hierarchic metadata structure can overcome some of those problems. I think we also need to recognize some of the practical limitations that face us. There are limits to the accuracy of the information that's in medical records; there are limits to the consistency in which people apply vocabulary terms. Computers can be completely precise in terms of mapping from this to that, but people will continue to have different conceptions of what a “grade II systolic murmur” is. Bill Stead: I agree with both answers, but I want to continue to clarify what we are talking about. We get in trouble because people use words to reduce concepts to something that we can manage in our heads. So we lump, and person A lumps differently from person B. So we are each a “legacy system,” and our information resources have grown from this starting point. I think we need to work at two ends of the spectrum. Whenever possible, capture data according to granular definitions. If we have an organism and we discover that it splits into six organisms, that's actually a very easy problem to solve, as you said. What you've got to do is say, “A is now B, C, and D and it mapped here.” That is straightforward. That's the end of the spectrum where we can stay granular. For example, never store a doctor and the doctor's service as one piece of information. At the other end of the spectrum, where the granular definitions are not obvious, do not try to classify the data. Instead, tag a “clump” of information with metadata. This tagging, together with increasingly sophisticated extraction techniques, will be used to approximate meaning. Over time, we will get to a complete set of coded data by working from the two ends. Mark Musen: I'm not sure that everything will ever be completely coded. Given the fact that the world is continuously changing, I don't think we can assume that Aristotle was correct that eventually there will be a classification that we will all accept. For example, I do not know whether gastric ulcer is an infectious disease or a gastrointestinal disease, and maybe it is both. As we continue to learn more about medicine and as our organizations change out from under us, I think we're going to be in the situation where the way we categorize the world is going to change. This is very hard stuff. Instead of working on the ultimate classification that will have all of the problems of the International Classification of Diseases, we need to build structures that not only allow us to enumerate the kinds of data that our programs operate on, but attempt as best as we can to enumerate the assumptions that we're making about our data and about the world. Then, as things change, we can, as human beings, try to update our ontologies. I think we have to be able to deal with changing worlds and with the fact that people and computers each need different views on the data, and that means different assumptions as well. Bill Hersh: To reiterate Mark's point, some people have heard this quote, that “perfect is the enemy of good.” We, especially us academic types, strive for perfection, but in reality the world is not perfect, and I don't know that everything will be perfectly coded. But we can reach compromises, such that we can code bits of information that enable us to do useful things. Bill Stead: I think human beings are each different, but we have an underlying genetic code that we are in the process of discovering. Next, we are going to have to work out the problem of going from genotype to phenotype. When I say that I think in the end things will be coded, I think we're going to discover something that is to information what DNA is to people. It will be a very granular base set of building blocks, which will be rolled up into concepts much as genes produce proteins. So I do not want to go to one ontology or one classification. Still, I like having ontologies, particularly ones that clearly represent the difference between themselves and the others. Member of the audience: I'd like to ask a question about capturing ontologies from multiple people. Imagine for a moment that knowledge freezes long enough for us to try to catch it. Do you have a vision of a tool that will allow multiple knowledge-domain people to act at once? To work out discrepancies in their visions? Mark Musen: Put differently, the question was how do we deal with the fact that there is no overarching ontology? How do we build the tools that will allow us to try to achieve consensus in ontologies? I think the answer to that question is that we do not know. I'm being a little bit facetious, but philosophers have been trying to deal with that problem for 2,000 to 3,000 years. I think you see two different approaches in the computer science community. You see the approach that Doug Lenat has taken. He is trying to create an ontology that he believes will provide all the knowledge that one needs to read the Encyclopaedia Britannica. Such an overarching ontology would need to capture most of human existence. The real problem, though, is how you ever validate the distinctions made in that ontology and have confidence that things have been captured in a way that is consistent and understandable? How do you record all the assumptions that you make while constructing the ontology? When you have concepts like “semi-tangible object” and “semi-intangible object,” it's very hard to know for sure whether what one records about those distinctions really makes sense. At the other end of the spectrum, you see people who really want a thousand flowers to bloom and who are not trying to achieve that kind of perfect alignment among views of the world. For example, the Knowledge Systems Laboratory at Stanford is trying to make constrained ontologies that deal with very narrow domains, so that the kinds of problems that you allude to do not happen, because the number of concepts in the ontology is relatively small. The answer lies somewhere between Doug Lenat's view of the world, that all we have to do is work hard enough and everything will fall into place, and the view that we can't possibly do this, so we have to have just a small number of constrained ontologies. We need to elucidate a set of principles that will provide the basis for tools that will help us try to, if not merge small ontologies, at least create the kinds of alignments that will allow us to bring them together in ways that make them useful. Randy Miller: One of the things that I learned from my mentor, Jack Myers, is that as an informatician, as opposed to a philosopher or a computer scientist, you do not need to represent everything. If you have a problem at hand, you represent it at a level that is tractable and doable. If you do what Doug Lenat's doing, you can spend your entire career representing stuff that is not ever going to be used in a real system, because there is no way to apply it. While that may sound harsh, the reality is that we do not know how to represent time, severity of finding, and severity of illness well at all, but we can still build systems that do diagnosis or a good job of making recommendations for therapy. So you do not have to capture the world in all its infinite detail. The trick is to understand what the critical information is and represent things at that level. Otherwise, you get mired in detail. Mark Musen: Let me underscore your last point. Doug Lenat actually felt pretty confident that his ontology covered all the areas that one would want to deal with, until last year, when HotBot contracted to use CYC as the basis for indexing Web pages. This contract showed, first of all, that ontologies have incredible commercial potential, but it also pointed out to Doug Lenat that there was a whole realm of human experience that was not well represented in the ontology. Specifically, there was a need to categorize different kinds of pornography, which Lenat had not thought about previously. Member of the audience: Health Level Seven's development of a set of reference information models is one of the major efforts for creating a structure for ontologies in the United States. Can you talk about how your organizations are participating in the development of that reference information model (RIM) and how you are using your academic experiences to contribute to that effort among providers, academics, and vendors? Bill Stead: Vanderbilt is an institutional member and a strong advocate of HL7. The central core of our communication subsystem uses HL7, and we build middle ware as needed to bridge between the core and legacy products. We have not put direct energy into the process for defining the reference information model. We use the HL7 model as a starting point, but we extend it as needed. In this way we incorporate it into immediate solutions to real problems, while providing useful information about future directions. Bill Hersh: None of us has been involved directly in that effort. However, our research into the nature of ontologies and the vocabulary projects such as the Cannon Grouping should useful to the effort. Mark Musen: I will just add that I think the vendor community is in the best position to work on ontology content, because they have the most direct connection with the needs of end users. I think that academicians need to follow this work very carefully. We are, we hope, in the best position to be developing the kinds of tools that will help us examine ontologies, relate them to each other, and allow them to evolve as our understanding of the world changes. Randy Miller: I have a slightly contrary view, partly out of ignorance about HL7 RIM. The key question is what problems it is trying to solve. That should drive what the content is. If you can state the problems it is going to be used to solve, then you can say whether it should clinically rich. In that case it will require lots of input from academic clinicians. If it is to solve the problem of interchange of data among vendors, then it needs vendor input. But until you explicitly state what it's going to be used for, just building it for the sake of building it is not useful. I know that the HL7 RIM is not being built that way. I am just saying that I think that's the way to address your question, to seek the specific purpose before giving an answer.

OBJECTIVES: To investigate the attitudes of students entering medical school toward the confidentiality of computerized medical records. DESIGN: First-year medical students at the Vanderbilt University School of Medicine responded to a series of questions about a hypothetic breach of patient's privacy through a computerized patient record system. MEASUREMENTS: The individual authors independently grouped the blinded responses according to whether they were consistent with then-current institutional policy. These preliminary groupings were discussed, and final categorizations were made by consensus. RESULTS: While most students had a sense of what was right and wrong in absolute terms, half the class suggested at least one course of action that was deemed to be inconsistent with institutional policies. CONCLUSIONS: The authors believe that medical schools should directly address ethical and legal issues related to the use of computers in clinical practice as an integral part of medical school curricula. Several teaching approaches can facilitate a greater awareness of the issues surrounding technology and medicine.

Synthesizing the state of the art from the published literature, this review assesses the basis for employing the Internet to support the information needs of primary care. The authors survey what has been published about the information needs of clinical practice, including primary care, and discuss currently available information resources potentially relevant to primary care. Potential methods of linking information needs with appropriate information resources are described in the context of previous classifications of clinical information needs. Also described is the role that existing terminology mapping systems, such as the National Library of Medicine's Unified Medical Language System, may play in representing and linking information needs to answers.

OBJECTIVE: A primary goal of the University of Pittsburgh's 1990-94 UMLS-sponsored effort was to develop and evaluate PostDoc (a lexical indexing system) and Pindex (a statistical indexing system) comparatively, and then in combination as a hybrid system. Each system takes as input a portion of the free text from a narrative part of a patient's electronic medical record and returns a list of suggested MeSH terms to use in formulating a Medline search that includes concepts in the text. This paper describes the systems and reports an evaluation. The intent is for this evaluation to serve as a step toward the eventual realization of systems that assist healthcare personnel in using the electronic medical record to construct patient-specific searches of Medline. DESIGN: The authors tested the performances of PostDoc, Pindex, and a hybrid system, using text taken from randomly selected clinical records, which were stratified to include six radiology reports, six pathology reports, and six discharge summaries. They identified concepts in the clinical records that might conceivably be used in performing a patient-specific Medline search. Each system was given the free text of each record as an input. The extent to which a system-derived list of MeSH terms captured the relevant concepts in these documents was determined based on blinded assessments by the authors. RESULTS: PostDoc output a mean of approximately 19 MeSH terms per report, which included about 40% of the relevant report concepts. Pindex output a mean of approximately 57 terms per report and captured about 45% of the relevant report concepts. A hybrid system captured approximately 66% of the relevant concepts and output about 71 terms per report. CONCLUSION: The outputs of PostDoc and Pindex are complementary in capturing MeSH terms from clinical free text. The results suggest possible approaches to reduce the number of terms output while maintaining the percentage of terms captured, including the use of UMLS semantic types to constrain the output list to contain only clinically relevant MeSH terms.

The UMLS Section in this issue of JAMIA is dedicated to the memory of Marsden Scott Blois, Jr., a pioneer in medical concept representation. Dr. Blois, an internationally recognized physician-investigator and medical informatician, made important contributions to melanoma research and to biomedical informatics during his distinguished and varied career. His keen insight in the early days of the UMLS project contributed significantly to the future direction and success of the project. This issue of JAMIA helps mark the tenth anniversary of the Unified Medical Language System (UMLS) project. From the beginning, the project has focused on overcoming the barriers users face when attempting to interact with computerized health information systems. The UMLS developers and their collaborators envisioned and then created a set of knowledge sources designed to support the development of sophisticated and accessible information systems. The UMLS knowledge sources have grown to now encompass the Metathesaurus (the largest of the knowledge sources), the Semantic Network, the Information Sources Map, and the SPECIALIST lexicon with its accompanying lexical programs. The Metathesaurus integrates more than 30 biomedical thesauri. The most recent release contains 331,756 concepts named by 739,439 different terms, including translations of some of the terminology into several other languages. Metathesaurus concepts are assigned semantic types from the Semantic Network, which, through its 135 semantic types and 51 relationships, provides a unifying semantic structure for the Metathesaurus terminology. The Information Sources Map is designed to serve as a resource for identifying databases and other information resources that are relevant to users' particular queries. The SPECIALIST lexicon is designed for use in natural language processing applications. The accompanying lexical programs work together with the lexicon (currently containing some 90,000 lexical records) in recognizing lexical variation in biomedical terminologies and texts. The UMLS data are distributed on CD-ROM, but, increasingly, researchers are gaining access by using the UMLS Knowledge Source Server, which provides flexible Internet-based access to all the knowledge sources.1 The papers included here necessarily represent only a small portion of the active research that has been conducted on the UMLS over the past ten years. For more detail, readers should examine the recently compiled bibliography of some 280 selected citations on the UMLS project.2 The current issue reflects a spectrum of publications that characterize UMLS research and development efforts. The contributions include a historical perspective on the UMLS project by its leaders3; a viewpoint paper by a prominent group not directly involved in core UMLS efforts on the role the ULMS now plays in addressing basic issues related to terminology development and use4; a paper on anatomic knowledge representation that proposes certain extensions to the UMLS Semantic Network and Metathesaurus5; a paper that describes semantically based methods to improve UMLS content quality and maintenance techniques6; and three reports on experiments applying the UMLS to specific clinical objectives.7–9 Two papers7, 9 represent international work on using and extending UMLS knowledge in health information systems. A decade of UMLS research has seen the development and testing of a rich set of continuously evolving knowledge sources. These knowledge sources are distributed regularly to the research community, a decision that was motivated by the belief that systems and tools improve through actual testing and use. Since early 1997, when the eighth annual release of the knowledge sources was announced, over 500 individuals and organizations worldwide have requested and received access to this eight edition. Since the first release of the knowledge sources in 1990, investigators have experimented with the UMLS data, and particularly its Metathesaurus, in a variety of application areas.2,3 The UMLS—which incorporates multiple terminologies, each designed for its own purposes and users—has, through its rich set of interrelationships, resulted in an evolving system that is greater than the sum of its component parts.4 As the coverage of the UMLS grows and tools to improve its update and maintenance flourish,6 it should find increased applicability in supporting research and education in the basic clinical sciences (e.g., Rosse et al.5), in bibliographic and information science research and products (e.g., the Internet Grateful Med Fact Sheet10), and in the clinical arena (e.g., Joubert et al.,7 Cooper and Miller,8 and Bodenreider et al.9)

OBJECTIVE: Refine the understanding of the desirable skills for health sciences librarians as a basis for developing a training program model that reflects the fundamental changes in health care delivery and information technology. DESIGN: A four-step needs assessment process: focus groups developed lists of desirable skills; the research team organized candidate skills into a taxonomy; a survey of a random sample of librarians and library users assessed perception of importance of individual skills; and the research team framed, as a unifying hypothesis, a training model. SURVEY METHODS: The survey was distributed to random samples of 150 librarians, stratified by type of library, and 150 library users, stratified by type of use. A non-randomized sample was obtained by mounting the survey on a World Wide Web server. The survey instrument included 96 distinct skills organized into 13 categories. Respondents rated the importance of each skill on a Likert scale and provided a separate ranking by identifying the ten most important skills for the profession. RESULTS: Among the participants, 51% of librarians and 36% of library users responded to the survey. All categories of skills were rated above the midpoint of priority on the Likert scale. All groups rated personality characteristics and skills as most important, with an understanding of the health sciences, education, and research being rated comparably to technical skills. CONCLUSIONS: Health sciences librarians need a new educational model that provides them with broad-based tools to discover new roles and new resources for acquiring individual skills as the need arises. A unifying training model would involve trainees in developing their learning plan in a way that promotes proactive inquiry and self-directed learning, and it would rotate the trainees through projects to provide skills and an understanding of end-user work processes.

In mid-1996, the FDA called for discussions on regulation of clinical software programs as medical devices. In response, a consortium of organizations dedicated to improving health care through information technology has developed recommendations for the responsible regulation and monitoring of clinical software systems by users, vendors, and regulatory agencies. Organizations assisting in development of recommendations, or endorsing the consortium position include the American Medical Informatics Association, the Computer-based Patient Record Institute, the Medical Library Association, the Association of Academic Health Sciences Libraries, the American Health Information Management Association, the American Nurses Association, the Center for Healthcare Information Management, and the American College of Physicians. The consortium proposes four categories of clinical system risks and four classes of measured monitoring and regulatory actions that can be applied strategically based on the level of risk in a given setting. The consortium recommends local oversight of clinical software systems, and adoption by healthcare information system developers of a code of good business practices. Budgetary and other constraints limit the type and number of systems that the FDA can regulate effectively. FDA regulation should exempt most clinical software systems and focus on those systems posing highest clinical risk, with limited opportunities for competent human intervention.

OBJECTIVE: To understand better the trade-offs of not incorporating explicit time in Quick Medical Reference (QMR), a diagnostic system in the domain of general internal medicine, along the dimensions of expressive power and diagnostic accuracy. DESIGN: The study was conducted in two phases. Phase I was a descriptive analysis of the temporal abstractions incorporated in QMR's terms. Phase II was a pseudo-prospective controlled experiment, measuring the effect of history and physical examination temporal content on the diagnostic accuracy of QMR. MEASUREMENTS: For each QMR finding that would fit our operational definition of temporal finding, several parameters describing the temporal nature of the finding were assessed, the most important ones being: temporal primitives, time units, temporal uncertainty, processes, and patterns. The history, physical examination, and initial laboratory results of 105 consecutive patients admitted to the Pittsburgh University Presbyterian Hospital were analyzed for temporal content and factors that could potentially influence diagnostic accuracy (these included: rareness of primary diagnosis, case length, uncertainty, spatial/causal information, and multiple diseases). RESULTS: 776 findings were identified as temporal. The authors developed an ontology describing the terms utilized by QMR developers to express temporal knowledge. The authors classified the temporal abstractions found in QMR in 116 temporal types, 11 temporal templates, and a temporal hierarchy. The odds of QMR's making a correct diagnosis in high temporal complexity cases is 0.7 the odds when the temporal complexity is lower, but this result is not statistically significant (95% confidence interval = 0.27-1.83). CONCLUSIONS: QMR contains extensive implicit time modeling. These results support the conclusion that the abstracted encoding of time in the medical knowledge of QMR does not induce a diagnostic performance penalty.

This case report describes development and testing of a method to extract clinical information stored in the Veterans Affairs (VA) Decentralized Hospital Computer System (DHCP) for the purpose of analyzing data about groups of patients. The authors used a microcomputer-based, structured query language (SQL)-compatible, relational database system to replicate a subset of the Nashville VA Hospital's DHCP patient database. This replicated database contained the complete current Nashville DHCP prescription, provider, patient, and drug data sets, and a subset of the laboratory data. A pilot project employed this replicated database to answer questions that might arise in drug-use evaluation, such as identification of cases of polypharmacy, suboptimal drug regimens, and inadequate laboratory monitoring of drug therapy. These database queries included as candidates for review all prescriptions for all outpatients. The queries demonstrated that specific drug-use events could be identified for any time interval represented in the replicated database.

Randolph A. Miller, MD; Evaluating Evaluations of Medical Diagnostic Systems, Journal of the American Medical Informatics Association, Volume 3, Issue 6, 1 Nove

Vanderbilt University Medical Center is implementing an Integrated Advanced Information Management System (IAIMS) using a fast-track approach. The elapsed time between start-up and completion of implementation will be 7.5 years. The Start-Up and Planning phases of the project are complete. The Implementation phase asks one question: How does an organization create an environment that redirects and coordinates a variety of individual activities so that they come together to provide an IAIMS? Four answers to this question are being tested. First, design resources to be "scalable"--i.e., capable of supporting enterprise-wide use. Second, provide information technology planning activities as ongoing core functions that direct local efforts. Third, design core infrastructure resources to be both reusable and expandable at the local level. Fourth, use milestones to measure progress toward selected endpoints to permit early refinement of plans and strategies.

OBJECTIVE: Evaluate the effects of long-term maintenance activities on existing portions of a large internal medicine knowledge base. DESIGN: Five physicians who were not among the original developers of the knowledge base independently updated a total of 15 QMR disease profiles; each updated submission was modified by a review of group serving as the "gold standard, " and the pre- and post-study versions of each updated disease profile were compared. MEASUREMENTS: Numbers and types of changes, defined as any difference between the original version and the final version of a disease profile; reason for each change; and bibliographic references cited by the physicians as supporting evidence. RESULTS: A total of 16% of all entries were modified by the updating process; up to 95% of the entries in a disease profile were affected. The two most common modifications were changes to the frequency of an entry, and creation of a new entry. Laboratory findings were affected much more often than were history, symptom, or physical exam findings. The dominant reason for changes was appearance of new evidence in the medical literature. The literature cited ranged from 1944 to the present. CONCLUSIONS: This study provides an evaluation of the rate of change within the QMR medical knowledge base due to long-term maintenance. The results show that this is a demanding activity that may profoundly affect certain portions of a knowledge base, and that different types of knowledge (e.g., simple laboratory vs expensive or invasive laboratory findings) are affected by the process in different ways.

Articles about medical diagnostic decision support (MDDS) systems often begin with a disclaimer such as, "despite many years of research and millions of dollars of expenditures on medical diagnostic systems, none is in widespread use at the present time." While this statement remains true in the sense that no single diagnostic system is in widespread use, it is misleading with regard to the state of the art of these systems. Diagnostic systems, many simple and some complex, are now ubiquitous, and research on MDDS systems is growing. The nature of MDDS systems has diversified over time. The prospects for adoption of large-scale diagnostic systems are better now than ever before, due to enthusiasm for implementation of the electronic medical record in academic, commercial, and primary care settings. Diagnostic decision support systems have become an established component of medical technology. This paper provides a review and a threaded bibliography for some of the important work on MDDS systems over the years from 1954 to 1993.