Automatic document categorization is an important research problem in Information Science and Natural Language Processing. Many applications, including Word Sense Disambiguation and Information Retrieval in large collections, can benefit from such categorization. This paper focuses on automatic categorization of documents from the biomedical literature into broad discipline-based categories. Two different systems are described and contrasted: CISMeF, which uses rules based on human indexing of the documents by the Medical Subject Headings(®) (MeSH(®)) controlled vocabulary in order to assign metaterms (MTs), and Journal Descriptor Indexing (JDI) based on human categorization of about 4,000 journals and statistical associations between journal descriptors (JDs) and textwords in the documents. We evaluate and compare the performance of these systems against a gold standard of humanly assigned categories for one hundred MEDLINE documents, using six measures selected from trec_eval. The results show that for five of the measures, performance is comparable, and for one measure, JDI is superior. We conclude that these results favor JDI, given the significantly greater intellectual overhead involved in human indexing and maintaining a rule base for mapping MeSH terms to MTs. We also note a JDI method that associates JDs with MeSH indexing rather than textwords, and it may be worthwhile to investigate whether this JDI method (statistical) and CISMeF (rule based) might be combined and then evaluated showing they are complementary to one another.

OBJECTIVE: To evaluate: (1) the effectiveness of wireless handheld computers for online information retrieval in clinical settings; (2) the role of MEDLINE in answering clinical questions raised at the point of care. DESIGN: A prospective single-cohort study: accompanying medical teams on teaching rounds, five internal medicine residents used and evaluated MD on Tap, an application for handheld computers, to seek answers in real time to clinical questions arising at the point of care. MEASUREMENTS: All transactions were stored by an intermediate server. Evaluators recorded clinical scenarios and questions, identified MEDLINE citations that answered the questions, and submitted daily and summative reports of their experience. A senior medical librarian corroborated the relevance of the selected citation to each scenario and question. RESULTS: Evaluators answered 68% of 363 background and foreground clinical questions during rounding sessions using a variety of MD on Tap features in an average session length of less than four minutes. The evaluator, the number and quality of query terms, the total number of citations found for a query, and the use of auto-spellcheck significantly contributed to the probability of query success. CONCLUSION: Handheld computers with Internet access are useful tools for healthcare providers to access MEDLINE in real time. MEDLINE citations can answer specific clinical questions when several medical terms are used to form a query. The MD on Tap application is an effective interface to MEDLINE in clinical settings, allowing clinicians to quickly find relevant citations.

An experiment was performed at the National Library of Medicine((R)) (NLM((R))) in word sense disambiguation (WSD) using the Journal Descriptor Indexing (JDI) methodology. The motivation is the need to solve the ambiguity problem confronting NLM's MetaMap system, which maps free text to terms corresponding to concepts in NLM's Unified Medical Language System((R)) (UMLS((R))) Metathesaurus((R)). If the text maps to more than one Metathesaurus concept at the same high confidence score, MetaMap has no way of knowing which concept is the correct mapping. We describe the JDI methodology, which is ultimately based on statistical associations between words in a training set of MEDLINE((R)) citations and a small set of journal descriptors (assigned by humans to journals per se) assumed to be inherited by the citations. JDI is the basis for selecting the best meaning that is correlated to UMLS semantic types (STs) assigned to ambiguous concepts in the Metathesaurus. For example, the ambiguity transport has two meanings: "Biological Transport" assigned the ST Cell Function and "Patient transport" assigned the ST Health Care Activity. A JDI-based methodology can analyze text containing transport and determine which ST receives a higher score for that text, which then returns the associated meaning, presumed to apply to the ambiguity itself. We then present an experiment in which a baseline disambiguation method was compared to four versions of JDI in disambiguating 45 ambiguous strings from NLM's WSD Test Collection. Overall average precision for the highest-scoring JDI version was 0.7873 compared to 0.2492 for the baseline method, and average precision for individual ambiguities was greater than 0.90 for 23 of them (51%), greater than 0.85 for 24 (53%), and greater than 0.65 for 35 (79%). On the basis of these results, we hope to improve performance of JDI and test its use in applications.

A new, fully automated approach for indexing documents is presented based on associating textwords in a training set of bibliographic citations with the indexing of journals. This journal-level indexing is in the form of a consistent, timely set of journal descriptors (JDs) indexing the individual journals themselves. This indexing is maintained in journal records in a serials authority database. The advantage of this novel approach is that the training set does not depend on previous manual indexing of hundreds of thousands of documents (i.e., any such indexing already in the training set is not used), but rather the relatively small intellectual effort of indexing at the journal level, usually a matter of a few thousand unique journals for which retrospective indexing to maintain consistency and currency may be feasible. If successful, JD indexing would provide topical categorization of documents outside the training set, i.e., journal articles, monographs, WEB documents, reports from the grey literature, etc., and therefore be applied in searching. Because JDs are quite general, corresponding to subject domains, their most probable use would be for improving or refining search results.

An overview of levels and approaches in the evaluation of knowledge-based systems is presented. There is a need for empirical studies using objective criteria in advance of completing the technical evaluation of such systems. A methodology for this type of evaluation developed for a particular knowledge-based indexing system is presented. It is suggested that the proposed study may serve as a model for the design of any evaluation in which the results of existing intellectual procedures are compared with results achieved when these procedures are aided by use of an appropriate expert system. © 1996 John Wiley & Sons, Inc.

This report discusses the Indexing Aid Project for conducting research in interactive knowledge-based indexing of the medical literature. After providing an overview and background, we describe and illustrate the Indexing Aid System using an extended example, highlighting the knowledge-based capabilities of the system, namely, inheritance and internal retrieval, enforcement of restrictions, and other functions implemented by procedural attachments, which are characteristic of frame-based knowledge representation languages. A feature which generates reports for evaluating the system is also shown. The paper concludes with discussion of the research plan. The project is part of the Automated Classification and Retrieval Program at the Lister Hill National Center for Biomedical Communications, the research and development arm of the National Library of Medicine.

This article describes the Indexing Aid Project for conducting research in the areas of knowledge representation and indexing for information retrieval in order to develop interactive knowledge-based systems for computer-assisted indexing of the periodical medical literature. The system uses an experimental frame-based knowledge representation language, FrameKit, implemented in Franz Lisp. The initial prototype is designed to interact with trained MEDLINE indexers who will be prompted to enter subject terms as slot values in filling in document-specific frame data structures that are derived from the knowledge-base frames. In addition, the automatic application of rules associated with the knowledge-base frames produces a set of Medical Subject Heading (MeSH) keyword indices to the document. Important features of the system are representation of explicit relationships through slots which express the relations; slot values, restrictions, and rules made available by inheritance through "is-a" hierarchies; slot values denoted by functions that retrieve values from other slots; and restrictions on slot values displayable during data entry.

This article addresses file maintenance of subject headings in the National Library of Medicine's MEDLINE, a bibliographic retrieval file which is indexed using Medical Subject Headings (MeSH), NLM's subject authority. The emphasis is class maintenance in which sets of records in MEDLINE are changed by maintenance actions which reflect yearly changes in MeSH. Specific types of maintenance action are described with examples to highlight problem areas. Class maintenance failures, checking class maintenance, and informing end-users of MeSH changes resulting in class maintenance are discussed. Technical constraints and feasibility are not analyzed in detail; rather, the approach is geared toward requirements and impact of maintenance from the standpoint of vocabulary specialists and end-users. The article concludes with a brief statement on defining file maintenance policy.