OBJECTIVES: The verification of biomedical ontologies is an arduous process that typically involves peer review by subject-matter experts. This work evaluated the ability of crowdsourcing methods to detect errors in SNOMED CT (Systematized Nomenclature of Medicine Clinical Terms) and to address the challenges of scalable ontology verification. METHODS: We developed a methodology to crowdsource ontology verification that uses micro-tasking combined with a Bayesian classifier. We then conducted a prospective study in which both the crowd and domain experts verified a subset of SNOMED CT comprising 200 taxonomic relationships. RESULTS: The crowd identified errors as well as any single expert at about one-quarter of the cost. The inter-rater agreement (κ) between the crowd and the experts was 0.58; the inter-rater agreement between experts themselves was 0.59, suggesting that the crowd is nearly indistinguishable from any one expert. Furthermore, the crowd identified 39 previously undiscovered, critical errors in SNOMED CT (eg, 'septic shock is a soft-tissue infection'). DISCUSSION: The results show that the crowd can indeed identify errors in SNOMED CT that experts also find, and the results suggest that our method will likely perform well on similar ontologies. The crowd may be particularly useful in situations where an expert is unavailable, budget is limited, or an ontology is too large for manual error checking. Finally, our results suggest that the online anonymous crowd could successfully complete other domain-specific tasks. CONCLUSIONS: We have demonstrated that the crowd can address the challenges of scalable ontology verification, completing not only intuitive, common-sense tasks, but also expert-level, knowledge-intensive tasks.

We present a domain specific ontology editor for viewing, updating and managing a clinical documentation knowledge base. The editor is designed to allow clinical content specialists, who do not have a working knowledge of OWL, Semantic Web technologies or knowledge engineering, to quickly generate ontologies that describe clinical documentation templates along with rich bi-directional mappings between these documentation template ontologies and biomedical domain ontologies. While the editor has been designed and implemented for a specific use-case, many of the novel design choices are applicable to more traditional ontology editing environments. Furthermore, we believe that the workflow that is promoted by the editing environment and the partitioning and arrangement of the ontologies that are compiled by the editor are applicable to more general scenarios where two sets of ontologies corresponding to different application sub-domains need to be edited side-by-side and mapped between using a set of binding ontologies.

The relation between an information entity and its referent can be described as a second-order statement, as long as the referent is a type. This is typical for medical discourse such as diagnostic statements in electronic health records (EHRs), which often express hypotheses or probability assertions about the existence of an instance of, e.g. a disease type. This paper presents several approximations using description logics and a query language, the entailments of which are checked against a reference standard. Their pros and cons are discussed in the light of formal ontology and logic.

The relationships between "ontologies", knowledge bases, and information models -- and correspondingly between OWL/Description Logics, frames and UML -- remains confusing to many developers. Understanding which to use when and developing effective hybrid systems that exploit the potential synergies requires clarifying key distinctions: between ontology, background knowledge, and information models; between axiombased and template-based systems; and between logical definitions and queries. As a step towards a more coordinated approach to knowledge-rich systems and a platform for incorporating additional technologies, we propose factoring systems into "ontology (narrow sense)", the rest of the "background knowledge base", and the "information model", with clear distinctions, mutual derivations and interfaces amongst them and clear understanding of the semantics and limitations of each.

SNOMED-CT is a large medical terminology based on description logic and mandated for use in the US, UK and several other countries. The hierarchies are known to contain many errors, but have so far proved difficult to analyse or quality assure. We present a series of methods and lessons learnt from experience in quality assuring a "module" of SNOMED for specific applications that we expect to generalize both to SNOMED as a whole and to other large ontologies. They feature a) dependence on domain exper-tise b) starting from classes selected for relevance to specific applications, c) tracing all errors to their root and verifying repairs by reclassification d) extraction of manageable-sized "modules"; e) mixed semantic and lexical criteria, and f) extensive use of scripting. They aim to reduce the cognitive load on experts by a) looking initially up-wards rather than downwards in the hierarchies, b) breaking up long lists of direct subclasses by introducing definitions for meaningful subcategories. Errors found range from simple mistakes to systematic errors in schemas. © 2011 ACM.

OBJECTIVES: (a) To determine the extent and range of errors and issues in the Systematised Nomenclature of Medicine-Clinical Terms (SNOMED CT) hierarchies as they affect two practical projects. (b) To determine the origin of issues raised and propose methods to address them. METHODS: The hierarchies for concepts in the Core Problem List Subset published by the Unified Medical Language System were examined for their appropriateness in two applications. Anomalies were traced to their source to determine whether they were simple local errors, systematic inferences propagated by SNOMED's classification process, or the result of problems with SNOMED's schemas. Conclusions were confirmed by showing that altering the root cause and reclassifying had the intended effects, and not others. MAIN RESULTS: Major problems were encountered, involving concepts central to medicine including myocardial infarction, diabetes, and hypertension. Most of the issues raised were systematic. Some exposed fundamental errors in SNOMED's schemas, particularly with regards to anatomy. In many cases, the root cause could only be identified and corrected with the aid of a classifier. LIMITATIONS: This is a preliminary 'experiment of opportunity.' The results are not exhaustive; nor is consensus on all points definitive. CONCLUSIONS: The SNOMED CT hierarchies cannot be relied upon in their present state in our applications. However, systematic quality assurance and correction are possible and practical but require sound techniques analogous to software engineering and combined lexical and semantic techniques. Until this is done, anyone using SNOMED codes should exercise caution. Errors in the hierarchies, or attempts to compensate for them, are likely to compromise interoperability and meaningful use.

Ontologies have been highly successful in applications involving annotation and data fusion. However, ontologies as the core of “Knowledge Driven Architectures” have not achieved the same influence as “Model Driven Architectures”, despite the fact that many biomedical applications require features that seem achievable only via ontological technologies – composition of descriptions, automatic classification and inference, and management of combinatorial explosions in many contexts. Our group adopted Knowledge Driven Architectures based on ontologies to address these problems in the early 1990s. In this paper we discuss first the use cases and requirements and then some of the requirements for more effective use of Knowledge Driven Architectures today: clearer separation of language and formal ontology, integration with contingent knowledge, richer and better distinguished annotations, higher order representations, integration with data models, and improved auxiliary structures to allow easy access and browsing by users.

We report on the development of a decision support ontology developed in the Web ontology language OWL-DL (description logic). The ontology is combined within a preoperative risk assessment software system with a DL reasoner in order to provide a number of clinical decision support functionalities, including risk assessment, recommended tests and recommended clinical precaution protocols.

A thorough documentation of a patient's medical history is widely recognised as providing good indicators of potential intraoperative and postoperative complications. As preoperative assessment can be time consuming, computer-based information collection systems (ICS) can help free up precious and limited resources, leaving clinicians with more time to fulfil their primary mission of administrating medical care. In addition, medical histories collected by ICSs have proved to be more accurate than traditional pen-and-paper questionnaires or face-to-face interviews. A challenge remains however in designing questionnaires which are general enough to suit a majority of patients, while at the same time, being able to capture critical individual information. In this paper, we propose a solution to this dilemma with a context-sensitive adaptive information collection system. The proposed method permits to iteratively capture finer-grained information with each successive step, should this information be relevant according to a questionnaire ontology. We argue that the method is robust, scalable and highly configurable. It results in questionnaires which are coherent and well structured and are able to capture enhanced patients' medical histories.

There has been major progress both in description logics and ontology design since SNOMED was originally developed. The emergence of the standard Web Ontology language in its latest revision, OWL 1.1 is leading to a rapid proliferation of tools. Combined with the increase in computing power in the past two decades, these developments mean that many of the restrictions that limited SNOMED's original formulation no longer need apply. We argue that many of the difficulties identified in SNOMED could be more easily dealt with using a more expressive language than that in which SNOMED was originally, and still is, formulated. The use of a more expressive language would bring major benefits including a uniform structure for context and negation. The result would be easier to use and would simplify developing software and formulating queries.

Current tools, techniques and methodologies for multi-user editing of semantic web ontologies are inadequate. The vast majority of ontologies are maintained by single individuals. However, single user access is increasingly becoming a bottleneck as these ontologies grow in size. We therefore suggest a technique and for locking segments of description logic ontologies for multi-user editing. This technique fits into a methodology for ontology editing in which multiple ontology engineers concurrently lock, extract, modify, error-check and re-merge individual segments of a large ontology. The technique aims to provide a pragmatic compromise between a very restrictive approach that might offer complete error protection but make useful multi-user interactions impossible and a wide-open anything-goes editing paradigm which offers little to no protection.

Abstract In a virtual organization environment, where services and data are provided and shared among organizations from different administrative domains and protected with dissimilar security policies and measures, there is a need for a flexible authentication framework that supports the use of various authentication methods and tokens. The authentication strengths derived from the authentication methods and tokens should be incorporated into an access‐control decision‐making process, so that more sensitive resources are available only to users authenticated with stronger methods. This paper reports our on‐going efforts in designing and implementing such a framework to facilitate multi‐level and multi‐factor adaptive authentication and authentication strength linked fine‐grained access control. The proof‐of‐concept prototype is designed and implemented in the Shibboleth and PERMIS infrastructures, which specifies protocols to federate authentication and authorization information and provides a policy‐driven, role‐based, access‐control decision‐making capability. Copyright © 2006 John Wiley & Sons, Ltd.

Ontologies are at the heart of the semantic web. They define the concepts and relationships that make global interoperability possible. However, as these ontologies grow in size they become more and more difficult to create, use, understand, maintain, transform and classify. We present and evaluate several algorithms for extracting relevant segments out of large description logic ontologies for the purposes of increasing tractability for both humans and computers. The segments are not mere fragments, but stand alone as ontologies in their own right. This technique takes advantage of the detailed semantics captured within an OWL ontology to produce highly relevant segments. The research was evaluated using the GALEN ontology of medical terms and procedures.

Modularity is a key requirement for large ontologies in order to achieve re-use, maintainability, and evolution. Mechanisms for 'normalisation' to achieve analogous aims are standard for databases. However, no similar notion of normalisation has yet emerged for ontologies. This paper proposes initial criteria for a two-step normalisation of ontologies implemented using OWL or related DL based formalisms. For the first - "ontological normalisation" - we accept Welty and Guarino's analysis. For the second - "implementation normalisation" - we propose an approach based on decomposing ("untangling") the ontology into independent disjoint skeleton taxonomies restricted to be simple trees, which can then be recombined using definitions and axioms to represent the relationships between them explicitly.

The GALEN programme has been developing medical ontologies collaboratively for nearly a decade. The ontologies are large and formulated in a specialised description logic, GRAIL. The programme is a broad collaboration of over a dozen groups, most with no prior experience of developing formal ontologies. The programme has developed a methodology for loosely coupled development using layers of intermediate representations, guidelines and tools which minimises training requirements for domain experts and effort by central knowledge engineers. Issues arise both from problems in formal representations and from the idiosyncrasies of the medical domain. Issues dealt with include 'tangled' taxonomies, part-whole and locative relationships, defaults and exceptions, semantic normalisation, and the difference between medical convention and strict logical criteria for correctness.

OBJECTIVE: The purpose of the study is twofold: 1) to explore the applicability of combinatorial terminologies as the basis for building enumerated classifications, and 2) to investigate the usefulness of formal terminological systems for performing such classification and for assisting in the refinement of both combinatorial terminologies and enumerated classifications. DESIGN: A formal model of the beta version of the International Classification for Nursing Practice (ICNP) was constructed in the compositional terminological language GRAIL (GALEN Representation and Integration Language). Terms drawn from the North American Nursing Diagnosis Association Taxonomy I (NANDA taxonomy) were mapped into the model and classified automatically using GALEN technology. MEASUREMENTS: The resulting generated hierarchy was compared with the NANDA taxonomy to assess coverage and accuracy of classification. RESULTS: In terms of coverage, in this study ICNP was able to capture 77 percent of NANDA terms using concepts drawn from five of its eight axes. Three axes-Body Site, Topology, and Frequency-were not needed. In terms of accuracy, where hierarchic relationships existed in the generated hierarchy or the NANDA taxonomy, or both, 6 were identical, 19 existed in the generated hierarchy alone (2 of these were considered suitable for incorporation into the NANDA taxonomy and 17 were considered inaccurate), and 23 appeared in the NANDA taxonomy alone (8 of these were considered suitable for incorporation into ICNP, 9 were considered inaccurate, and 6 reflected different, equally valid perspectives). Sixty terms appeared at the top level, with no indenting, in both the generated hierarchy and the NANDA taxonomy. CONCLUSIONS: With appropriate refinement, combinatorial terminologies such as ICNP have the potential to provide a useful foundation for representing enumerated classifications such as NANDA. Technologies such as GALEN make possible the process of building automatically enumerated classifications while providing a useful means of validating and refining both combinatorial terminologies and enumerated classifications.

OBJECTIVE: The purpose of the study is to explore the use of formal systems to model nursing terminology. DESIGN: GRAIL is a formal, compositional terminologic language, closely related to frame-based systems and conceptual graphs, which allows concepts to be formed from atomic-level primitives and automatically classified in a multiple hierarchy. A formal model of the alpha version of the International Classification for Nursing Practice (ICNP) classification of nursing interventions was constructed in GRAIL. MEASUREMENTS: The model was analyzed for completeness, coherence, clarity, expressiveness, usefulness, and maintainability. RESULTS: GRAIL is capable of representing the complete set of atomic-level concepts within the ICNP as well as certain cross-mappings to other vocabularies. It also has the potential to represent many more concepts, to an arbitrary level of detail. CONCLUSIONS: Formal systems such as GRAIL can overcome many of the difficulties associated with traditional nursing vocabularies without restricting the level of detail needed to describe nursing care.

A common language, or terminology, for representing what clinicians have said and done is an important requirement for individual clinical systems, and it is a pre-requisite for integrating disparate applications in a distributed telematic healthcare environment. Formal representations based on description logics or closely related formalisms are increasingly used for representing medical terminologies. GALEN's experience in using one such formalism raises two major issues, as follows: how to make ontologies based on description logics easy to use and understand for both clinicians and applications developers; what features are required of the ontology and description logic if they are to achieve their aims. Based on our experience we put forward four contentions: two relating to each of these two issues, as follows: that natural language generation is essential to make a description logic based ontology accessible to users; that the description logic based ontology should be treated as an "assembly language" and accessed via "intermediate representations" oriented to users and "perspectives" adapting it to specific applications; that independence and reuse are best supported by partitioning the subsumption hierarchy of elementary concepts into orthogonal taxonomies, each of which forms a pure tree in which the branches at each level are disjoint but nonexhaustive subconcepts of the parent concept; that the expressivity of the description logic must include support for transitive relations despite the computational cost, and that this computational cost is acceptable in practice. The authors argue that these features will be necessary, though by no means sufficient, for the development of any large reusable ontology for medicine.

OBJECTIVES: To investigate the issues raised in applying a preliminary version of the GALEN compositional concept reference (CORE) model to a series of radiographic reports, and to demonstrate that the same underlying concept model could be used in conjunction with both a detailed, fine-grained model of medical records based on that used in the PEN&PAD project and with other more conventional medical-record models. DESIGN: Following analysis and representation of concepts from a set of reports, a single report was taken as a "case study." This report was analyzed in detail in its entirety and represented using each of the medical-record models. RESULTS: The reports were successfully represented within the limits of the study, but a number of significant issues were raised. CONCLUSION: The compositional approach plus the PEN&PAD medical-record model allowed detailed information in the radiographic report to be represented, including information about the inferences and the clinical process. The resulting representation was large, and more compact representations may be necessary for some systems. Alternative encapsulations of the information as might be used in such systems were successfully prepared. The compositional approach avoided many issues that often cause controversy in the design of traditional coding and classification systems, but it raised other issues, including the handling of ambiguity and underspecification, linkage to information not explicitly present in the report, and questions concerning the focus of individual concepts. All work is preliminary and definitive conclusions await further studies and systematic evaluation.

It is argued that current clinical information systems incorporate oversimplistic, prescriptive data models that are not faithful to clinicians' observations. A non-prescriptive descriptive semantic formalism, Structured Meta Knowledge (SMK), which unifies a terminological knowledge base with controlled assertional capabilities with the medical record and supports the semantic control necessitated by such an approach, is proposed. The three-layer model of categories, individuals, and occurrences described is more appropriate to medical applications than the two layers of classes and instances. The application of SMK in predictive data entry is considered.>

The goal of the PEN&PAD project is to design and develop a useful and usable medical workstation for day–to–day use in patient care. The project has adopted a user centred approach and direct observations of doctors, participative design and Formative Evaluation have therefore been an integral part of the process of software development. Indeed, doctors have been involved from the earliest stages of the project. The project has focussed on British General Practitioners, but the methods which have been evolved are general. This paper describes the strategy by which doctors can be involved in the successful design and development of a medical workstation.