BACKGROUND: Chatbots are increasingly used to deliver health education, patient engagement, and access to healthcare services. GARDE-Chat is an open-source platform designed to facilitate the development, deployment, and dissemination of chatbot-based digital health interventions across different domains and settings. MATERIALS AND METHODS: GARDE-Chat was developed through an iterative process informed by real-world use cases to guide prioritization of key features. The tool was developed as an open-source platform to promote collaboration, broad dissemination, and impact across research and clinical domains. RESULTS: GARDE-Chat's main features include (1) a visual authoring interface that allows non-programmers to design chatbots; (2) support for scripted, large language model (LLM)-based and hybrid chatbots; (3) capacity to share chatbots with researchers and institutions; (4) integration with external applications and data sources such as electronic health records and REDCap; (5) delivery via web browsers or text messaging; and (6) detailed audit log supporting analyses of chatbot user interactions. Since its first release in July 2022, GARDE-Chat has supported the development of chatbot-based interventions tested in multiple studies, including large pragmatic clinical trials addressing topics such as genetic testing, COVID-19 testing, tobacco cessation, and cancer screening. DISCUSSION: Ongoing challenges include the effort required for developing chatbot scripts, ensuring safe use of LLMs, and integrating with clinical systems. CONCLUSION: GARDE-Chat is a generalizable platform for creating, implementing, and disseminating scalable chatbot-based population health interventions. It has been validated in several studies, and it is available to researchers and healthcare systems through an open-source mechanism.

MOTIVATION: Software is vital for the advancement of biology and medicine. Impact evaluations of scientific software have primarily emphasized traditional citation metrics of associated papers, despite these metrics inadequately capturing the dynamic picture of impact and despite challenges with improper citation. RESULTS: To understand how software developers evaluate their tools, we conducted a survey of participants in the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We found that although developers realize the value of more extensive metric collection, they find a lack of funding and time hindering. We also investigated software among this community for how often infrastructure that supports more nontraditional metrics were implemented and how this impacted rates of papers describing usage of the software. We found that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seemed to be associated with increased mention rates. Analysing more diverse metrics can enable developers to better understand user engagement, justify continued funding, identify novel use cases, pinpoint improvement areas, and ultimately amplify their software's impact. Challenges are associated, including distorted or misleading metrics, as well as ethical and security concerns. More attention to nuances involved in capturing impact across the spectrum of biomedical software is needed. For funders and developers, we outline guidance based on experience from our community. By considering how we evaluate software, we can empower developers to create tools that more effectively accelerate biological and medical research progress. AVAILABILITY AND IMPLEMENTATION: More information about the analysis, as well as access to data and code is available at https://github.com/fhdsl/ITCR_Metrics_manuscript_website.

OBJECTIVES: Patient care using genetics presents complex challenges. Clinical decision support (CDS) tools are a potential solution because they provide patient-specific risk assessments and/or recommendations at the point of care. This systematic review evaluated the literature on CDS systems which have been implemented to support genetically guided precision medicine (GPM). MATERIALS AND METHODS: A comprehensive search was conducted in MEDLINE and Embase, encompassing January 1, 2011-March 14, 2023. The review included primary English peer-reviewed research articles studying humans, focused on the use of computers to guide clinical decision-making and delivering genetically guided, patient-specific assessments, and/or recommendations to healthcare providers and/or patients. RESULTS: The search yielded 3832 unique articles. After screening, 41 articles were identified that met the inclusion criteria. Alerts and reminders were the most common form of CDS used. About 27 systems were integrated with the electronic health record; 2 of those used standards-based approaches for genomic data transfer. Three studies used a framework to analyze the implementation strategy. DISCUSSION: Findings include limited use of standards-based approaches for genomic data transfer, system evaluations that do not employ formal frameworks, and inconsistencies in the methodologies used to assess genetic CDS systems and their impact on patient outcomes. CONCLUSION: We recommend that future research on CDS system implementation for genetically GPM should focus on implementing more CDS systems, utilization of standards-based approaches, user-centered design, exploration of alternative forms of CDS interventions, and use of formal frameworks to systematically evaluate genetic CDS systems and their effects on patient care.

OBJECTIVES: To enhance the Business Process Management (BPM)+ Healthcare language portfolio by incorporating knowledge types not previously covered and to improve the overall effectiveness and expressiveness of the suite to improve Clinical Knowledge Interoperability. METHODS: We used the BPM+ Health and Object Management Group (OMG) standards development methodology to develop new languages, following a gap analysis between existing BPM+ Health languages and clinical practice guideline knowledge types. Proposal requests were developed based on these requirements, and submission teams were formed to respond to them. The resulting proposals were submitted to OMG for ratification. RESULTS: The BPM+ Health family of languages, which initially consisted of the Business Process Model and Notation, Decision Model and Notation, and Case Model and Notation, was expanded by adding 5 new language standards through the OMG. These include Pedigree and Provenance Model and Notation for expressing epistemic knowledge, Knowledge Package Model and Notation for supporting packaging knowledge, Shared Data Model and Notation for expressing ontic knowledge, Party Model and Notation for representing entities and organizations, and Specification Common Elements, a language providing a standard abstract and reusable library that underpins the 4 new languages. DISCUSSION AND CONCLUSION: In this effort, we adopted a strategy of separation of concerns to promote a portfolio of domain-agnostic, independent, but integrated domain-specific languages for authoring medical knowledge. This strategy is a practical and effective approach to expressing complex medical knowledge. These new domain-specific languages offer various knowledge-type options for clinical knowledge authors to choose from without potentially adding unnecessary overhead or complexity.

OBJECTIVE: Obtain clinicians' perspectives on early warning scores (EWS) use within context of clinical cases. MATERIAL AND METHODS: We developed cases mimicking sepsis situations. De-identified data, synthesized physician notes, and EWS representing deterioration risk were displayed in a simulated EHR for analysis. Twelve clinicians participated in semi-structured interviews to ascertain perspectives across four domains: (1) Familiarity with and understanding of artificial intelligence (AI), prediction models and risk scores; (2) Clinical reasoning processes; (3) Impression and response to EWS; and (4) Interface design. Transcripts were coded and analyzed using content and thematic analysis. RESULTS: Analysis revealed clinicians have experience but limited AI and prediction/risk modeling understanding. Case assessments were primarily based on clinical data. EWS went unmentioned during initial case analysis; although when prompted to comment on it, they discussed it in subsequent cases. Clinicians were unsure how to interpret or apply the EWS, and desired evidence on its derivation and validation. Design recommendations centered around EWS display in multi-patient lists for triage, and EWS trends within the patient record. Themes included a "Trust but Verify" approach to AI and early warning information, dichotomy that EWS is helpful for triage yet has disproportional signal-to-high noise ratio, and action driven by clinical judgment, not the EWS. CONCLUSIONS: Clinicians were unsure of how to apply EWS, acted on clinical data, desired score composition and validation information, and felt EWS was most useful when embedded in multi-patient views. Systems providing interactive visualization may facilitate EWS transparency and increase confidence in AI-generated information.

OBJECTIVE: This study aimed to 1) investigate algorithm enhancements for identifying patients eligible for genetic testing of hereditary cancer syndromes using family history data from electronic health records (EHRs); and 2) assess their impact on relative differences across sex, race, ethnicity, and language preference. MATERIALS AND METHODS: The study used EHR data from a tertiary academic medical center. A baseline rule-base algorithm, relying on structured family history data (structured data; SD), was enhanced using a natural language processing (NLP) component and a relaxed criteria algorithm (partial match [PM]). The identification rates and differences were analyzed considering sex, race, ethnicity, and language preference. RESULTS: Among 120,007 patients aged 25-60, detection rate differences were found across all groups using the SD (all P < 0.001). Both enhancements increased identification rates; NLP led to a 1.9 % increase and the relaxed criteria algorithm (PM) led to an 18.5 % increase (both P < 0.001). Combining SD with NLP and PM yielded a 20.4 % increase (P < 0.001). Similar increases were observed within subgroups. Relative differences persisted across most categories for the enhanced algorithms, with disproportionately higher identification of patients who are White, Female, non-Hispanic, and whose preferred language is English. CONCLUSION: Algorithm enhancements increased identification rates for patients eligible for genetic testing of hereditary cancer syndromes, regardless of sex, race, ethnicity, and language preference. However, differences in identification rates persisted, emphasizing the need for additional strategies to reduce disparities such as addressing underlying biases in EHR family health information and selectively applying algorithm enhancements for disadvantaged populations. Systematic assessment of differences in algorithm performance across population subgroups should be incorporated into algorithm development processes.

Embedded pragmatic clinical trials (ePCTs) play a vital role in addressing current population health problems, and their use of electronic health record (EHR) systems promises efficiencies that will increase the speed and volume of relevant and generalizable research. However, as the number of ePCTs using EHR-derived data grows, so does the risk that research will become more vulnerable to biases due to differences in data capture and access to care for different subsets of the population, thereby propagating inequities in health and the healthcare system. We identify 3 challenges-incomplete and variable capture of data on social determinants of health, lack of representation of vulnerable populations that do not access or receive treatment, and data loss due to variable use of technology-that exacerbate bias when working with EHR data and offer recommendations and examples of ways to actively mitigate bias.

OBJECTIVE: Surveillance algorithms that predict patient decompensation are increasingly integrated with clinical workflows to help identify patients at risk of in-hospital deterioration. This scoping review aimed to identify the design features of the information displays, the types of algorithm that drive the display, and the effect of these displays on process and patient outcomes. MATERIALS AND METHODS: The scoping review followed Arksey and O'Malley's framework. Five databases were searched with dates between January 1, 2009 and January 26, 2022. Inclusion criteria were: participants-clinicians in inpatient settings; concepts-intervention as deterioration information displays that leveraged automated AI algorithms; comparison as usual care or alternative displays; outcomes as clinical, workflow process, and usability outcomes; and context as simulated or real-world in-hospital settings in any country. Screening, full-text review, and data extraction were reviewed independently by 2 researchers in each step. Display categories were identified inductively through consensus. RESULTS: Of 14 575 articles, 64 were included in the review, describing 61 unique displays. Forty-one displays were designed for specific deteriorations (eg, sepsis), 24 provided simple alerts (ie, text-based prompts without relevant patient data), 48 leveraged well-accepted score-based algorithms, and 47 included nurses as the target users. Only 1 out of the 10 randomized controlled trials reported a significant effect on the primary outcome. CONCLUSIONS: Despite significant advancements in surveillance algorithms, most information displays continue to leverage well-understood, well-accepted score-based algorithms. Users' trust, algorithmic transparency, and workflow integration are significant hurdles to adopting new algorithms into effective decision support tools.

INTRODUCTION: The use and interoperability of clinical knowledge starts with the quality of the formalism utilized to express medical expertise. However, a crucial challenge is that existing formalisms are often suboptimal, lacking the fidelity to represent complex knowledge thoroughly and concisely. Often this leads to difficulties when seeking to unambiguously capture, share, and implement the knowledge for care improvement in clinical information systems used by providers and patients. OBJECTIVES: To provide a systematic method to address some of the complexities of knowledge composition and interoperability related to standards-based representational formalisms of medical knowledge. METHODS: Several cross-industry (Healthcare, Linguistics, System Engineering, Standards Development, and Knowledge Engineering) frameworks were synthesized into a proposed reference knowledge framework. The framework utilizes IEEE 42010, the MetaObject Facility, the Semantic Triangle, an Ontology Framework, and the Domain and Comprehensibility Appropriateness criteria. The steps taken were: 1) identify foundational cross-industry frameworks, 2) select architecture description method, 3) define life cycle viewpoints, 4) define representation and knowledge viewpoints, 5) define relationships between neighboring viewpoints, and 6) establish characteristic definitions of the relationships between components. System engineering principles applied included separation of concerns, cohesion, and loose coupling. RESULTS: A "Multilayer Metamodel for Representation and Knowledge" (M*R/K) reference framework was defined. It provides a standard vocabulary for organizing and articulating medical knowledge curation perspectives, concepts, and relationships across the artifacts created during the life cycle of language creation, authoring medical knowledge, and knowledge implementation in clinical information systems such as electronic health records (EHR). CONCLUSION: M*R/K provides a systematic means to address some of the complexities of knowledge composition and interoperability related to medical knowledge representations used in diverse standards. The framework may be used to guide the development, assessment, and coordinated use of knowledge representation formalisms. M*R/K could promote the alignment and aggregated use of distinct domain-specific languages in composite knowledge artifacts such as clinical practice guidelines (CPGs).

Population health management (PHM) is an important approach to promote wellness and deliver health care to targeted individuals who meet criteria for preventive measures or treatment. A critical component for any PHM program is a data analytics platform that can target those eligible individuals. OBJECTIVE: The aim of this study was to design and implement a scalable standards-based clinical decision support (CDS) approach to identify patient cohorts for PHM and maximize opportunities for multi-site dissemination. MATERIALS AND METHODS: An architecture was established to support bidirectional data exchanges between heterogeneous electronic health record (EHR) data sources, PHM systems, and CDS components. HL7 Fast Healthcare Interoperability Resources and CDS Hooks were used to facilitate interoperability and dissemination. The approach was validated by deploying the platform at multiple sites to identify patients who meet the criteria for genetic evaluation of familial cancer. RESULTS: The Genetic Cancer Risk Detector (GARDE) platform was created and is comprised of four components: (1) an open-source CDS Hooks server for computing patient eligibility for PHM cohorts, (2) an open-source Population Coordinator that processes GARDE requests and communicates results to a PHM system, (3) an EHR Patient Data Repository, and (4) EHR PHM Tools to manage patients and perform outreach functions. Site-specific deployments were performed on onsite virtual machines and cloud-based Amazon Web Services. DISCUSSION: GARDE's component architecture establishes generalizable standards-based methods for computing PHM cohorts. Replicating deployments using one of the established deployment methods requires minimal local customization. Most of the deployment effort was related to obtaining site-specific information technology governance approvals.

OBJECTIVE: The US Preventive Services Task Force (USPSTF) requires the estimation of lifetime pack-years to determine lung cancer screening eligibility. Leading electronic health record (EHR) vendors calculate pack-years using only the most recently recorded smoking data. The objective was to characterize EHR smoking data issues and to propose an approach to addressing these issues using longitudinal smoking data. MATERIALS AND METHODS: In this cross-sectional study, we evaluated 16 874 current or former smokers who met USPSTF age criteria for screening (50-80 years old), had no prior lung cancer diagnosis, and were seen in 2020 at an academic health system using the Epic® EHR. We described and quantified issues in the smoking data. We then estimated how many additional potentially eligible patients could be identified using longitudinal data. The approach was verified through manual review of records from 100 subjects. RESULTS: Over 80% of evaluated records had inaccuracies, including missing packs-per-day or years-smoked (42.7%), outdated data (25.1%), missing years-quit (17.4%), and a recent change in packs-per-day resulting in inaccurate lifetime pack-years estimation (16.9%). Addressing these issues by using longitudinal data enabled the identification of 49.4% more patients potentially eligible for lung cancer screening (P < .001). DISCUSSION: Missing, outdated, and inaccurate smoking data in the EHR are important barriers to effective lung cancer screening. Data collection and analysis strategies that reflect changes in smoking habits over time could improve the identification of patients eligible for screening. CONCLUSION: The use of longitudinal EHR smoking data could improve lung cancer screening.

OBJECTIVE: To evaluate the potential for machine learning to predict medication alerts that might be ignored by a user, and intelligently filter out those alerts from the user's view. MATERIALS AND METHODS: We identified features (eg, patient and provider characteristics) proposed to modulate user responses to medication alerts through the literature; these features were then refined through expert review. Models were developed using rule-based and machine learning techniques (logistic regression, random forest, support vector machine, neural network, and LightGBM). We collected log data on alerts shown to users throughout 2019 at University of Utah Health. We sought to maximize precision while maintaining a false-negative rate <0.01, a threshold predefined through discussion with physicians and pharmacists. We developed models while maintaining a sensitivity of 0.99. Two null hypotheses were developed: H1-there is no difference in precision among prediction models; and H2-the removal of any feature category does not change precision. RESULTS: A total of 3,481,634 medication alerts with 751 features were evaluated. With sensitivity fixed at 0.99, LightGBM achieved the highest precision of 0.192 and less than 0.01 for the pre-defined maximal false-negative rate by subject-matter experts (H1) (P < 0.001). This model could reduce alert volume by 54.1%. We removed different combinations of features (H2) and found that not all features significantly contributed to precision. Removing medication order features (eg, dosage) most significantly decreased precision (-0.147, P = 0.001). CONCLUSIONS: Machine learning potentially enables the intelligent filtering of medication alerts.

OBJECTIVE: The Genomic Medicine Working Group of the National Advisory Council for Human Genome Research virtually hosted its 13th genomic medicine meeting titled "Developing a Clinical Genomic Informatics Research Agenda". The meeting's goal was to articulate a research strategy to develop Genomics-based Clinical Informatics Tools and Resources (GCIT) to improve the detection, treatment, and reporting of genetic disorders in clinical settings. MATERIALS AND METHODS: Experts from government agencies, the private sector, and academia in genomic medicine and clinical informatics were invited to address the meeting's goals. Invitees were also asked to complete a survey to assess important considerations needed to develop a genomic-based clinical informatics research strategy. RESULTS: Outcomes from the meeting included identifying short-term research needs, such as designing and implementing standards-based interfaces between laboratory information systems and electronic health records, as well as long-term projects, such as identifying and addressing barriers related to the establishment and implementation of genomic data exchange systems that, in turn, the research community could help address. DISCUSSION: Discussions centered on identifying gaps and barriers that impede the use of GCIT in genomic medicine. Emergent themes from the meeting included developing an implementation science framework, defining a value proposition for all stakeholders, fostering engagement with patients and partners to develop applications under patient control, promoting the use of relevant clinical workflows in research, and lowering related barriers to regulatory processes. Another key theme was recognizing pervasive biases in data and information systems, algorithms, access, value, and knowledge repositories and identifying ways to resolve them.

OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

OBJECTIVE: In many cases, genetic testing labs provide their test reports as portable document format files or scanned images, which limits the availability of the contained information to advanced informatics solutions, such as automated clinical decision support systems. One of the promising standards that aims to address this limitation is Health Level Seven International (HL7) Fast Healthcare Interoperability Resources Clinical Genomics Implementation Guide-Release 1 (FHIR CG IG STU1). This study aims to identify various data content of some genetic lab test reports and map them to FHIR CG IG specification to assess its coverage and to provide some suggestions for standard development and implementation. MATERIALS AND METHODS: We analyzed sample reports of 4 genetic tests and relevant professional reporting guidelines to identify their key data elements (KDEs) that were then mapped to FHIR CG IG. RESULTS: We identified 36 common KDEs among the analyzed genetic test reports, in addition to other unique KDEs for each genetic test. Relevant suggestions were made to guide the standard implementation and development. DISCUSSION AND CONCLUSION: The FHIR CG IG covers the majority of the identified KDEs. However, we suggested some FHIR extensions that might better represent some KDEs. These extensions may be relevant to FHIR implementations or future FHIR updates.The FHIR CG IG is an excellent step toward the interoperability of genetic lab test reports. However, it is a work-in-progress that needs informative and continuous input from the clinical genetics' community, specifically professional organizations, systems implementers, and genetic knowledgebase providers.

OBJECTIVE: The purpose of the study was to explore the theoretical underpinnings of effective clinical decision support (CDS) factors using the comparative effectiveness results. MATERIALS AND METHODS: We leveraged search results from a previous systematic literature review and updated the search to screen articles published from January 2017 to January 2020. We included randomized controlled trials and cluster randomized controlled trials that compared a CDS intervention with and without specific factors. We used random effects meta-regression procedures to analyze clinician behavior for the aggregate effects. The theoretical model was the Unified Theory of Acceptance and Use of Technology (UTAUT) model with motivational control. RESULTS: Thirty-four studies were included. The meta-regression models identified the importance of effort expectancy (estimated coefficient = -0.162; P = .0003); facilitating conditions (estimated coefficient = 0.094; P = .013); and performance expectancy with motivational control (estimated coefficient = 1.029; P = .022). Each of these factors created a significant impact on clinician behavior. The meta-regression model with the multivariate analysis explained a large amount of the heterogeneity across studies (R2 = 88.32%). DISCUSSION: Three positive factors were identified: low effort to use, low controllability, and providing more infrastructure and implementation strategies to support the CDS. The multivariate analysis suggests that passive CDS could be effective if users believe the CDS is useful and/or social expectations to use the CDS intervention exist. CONCLUSIONS: Overall, a modified UTAUT model that includes motivational control is an appropriate model to understand psychological factors associated with CDS effectiveness and to guide CDS design, implementation, and optimization.

OBJECTIVE: We identified challenges and solutions to using electronic health record (EHR) systems for the design and conduct of pragmatic research. MATERIALS AND METHODS: Since 2012, the Health Care Systems Research Collaboratory has served as the resource coordinating center for 21 pragmatic clinical trial demonstration projects. The EHR Core working group invited these demonstration projects to complete a written semistructured survey and used an inductive approach to review responses and identify EHR-related challenges and suggested EHR enhancements. RESULTS: We received survey responses from 20 projects and identified 21 challenges that fell into 6 broad themes: (1) inadequate collection of patient-reported outcome data, (2) lack of structured data collection, (3) data standardization, (4) resources to support customization of EHRs, (5) difficulties aggregating data across sites, and (6) accessing EHR data. DISCUSSION: Based on these findings, we formulated 6 prerequisites for PCTs that would enable the conduct of pragmatic research: (1) integrate the collection of patient-centered data into EHR systems, (2) facilitate structured research data collection by leveraging standard EHR functions, usable interfaces, and standard workflows, (3) support the creation of high-quality research data by using standards, (4) ensure adequate IT staff to support embedded research, (5) create aggregate, multidata type resources for multisite trials, and (6) create re-usable and automated queries. CONCLUSION: We are hopeful our collection of specific EHR challenges and research needs will drive health system leaders, policymakers, and EHR designers to support these suggestions to improve our national capacity for generating real-world evidence.

OBJECTIVE: To facilitate the development of standards-based clinical decision support (CDS) systems, we review the current set of CDS standards that are based on Health Level Seven International Fast Healthcare Interoperability Resources (FHIR). Widespread adoption of these standards may help reduce healthcare variability, improve healthcare quality, and improve patient safety. TARGET AUDIENCE: This tutorial is designed for the broad informatics community, some of whom may be unfamiliar with the current, FHIR-based CDS standards. SCOPE: This tutorial covers the following standards: Arden Syntax (using FHIR as the data model), Clinical Quality Language, FHIR Clinical Reasoning, SMART on FHIR, and CDS Hooks. Detailed descriptions and selected examples are provided.

The psychology of motivation can help us understand the impact of electronic health records (EHRs) on clinician burnout both directly and indirectly. Informatics approaches to EHR usability tend to focus on the extrinsic motivation associated with successful completion of clearly defined tasks in clinical workflows. Intrinsic motivation, which includes the need for autonomy, sense-making, creativity, connectedness, and mastery is not well supported by current designs and workflows. This piece examines existing research on the importance of 3 psychological drives in relation to healthcare technology: goal-based decision-making, sense-making, and agency/autonomy. Because these motives are ubiquitous, foundational to human functioning, automatic, and unconscious, they may be overlooked in technological interventions. The results are increased cognitive load, emotional distress, and unfulfilling workplace environments. Ultimately, we hope to stimulate new research on EHR design focused on expanding functionality to support intrinsic motivation, which, in turn, would decrease burnout and improve care.

Cancer survivorship has traditionally received little research attention although it is associated with a variety of long-term consequences and also many other comorbidities. There is an urgent need to increase research on this area, and the secondary use of healthcare data has the potential to provide valuable insights on survivors' health trajectories. However, cancer survivors' data is often stored in silos and collected inconsistently. In this study we present CASIDE, an interoperable data model for cancer survivorship information that aims to accelerate the secondary use of healthcare data and data sharing across institutions. It is designed to provide a holistic view of the cancer survivor, taking into account not just the clinical data but also the patient's own perspective, and is built upon the emerging Health Level Seven (HL7) Fast Healthcare Interoperability Resources (FHIR) standard. Advantages of adopting FHIR and challenges in information modelling using this standard are discussed. CASIDE is a generalizable approach that is already being used as a support tool for the development of downstream applications to support clinical decision making and can contribute to translational collaborative research on cancer survivorship.

OBJECTIVE: The objective of this project was to enable poison control center (PCC) participation in standards-based health information exchange (HIE). Previously, PCC participation was not possible due to software noncompliance with HIE standards, lack of informatics infrastructure, and the need to integrate HIE processes into workflow. MATERIALS AND METHODS: We adapted the Health Level Seven Consolidated Clinical Document Architecture (C-CDA) consultation note for the PCC use case. We used rapid prototyping to determine requirements for an HIE dashboard for use by PCCs and developed software called SNOWHITE that enables poison center HIE in tandem with a poisoning information system. RESULTS: We successfully implemented the process and software at the PCC and began sending outbound C-CDAs from the Utah PCC on February 15, 2017; we began receiving inbound C-CDAs on October 30, 2018. DISCUSSION: With the creation of SNOWHITE and initiation of an HIE process for sending outgoing C-CDA consultation notes from the Utah Poison Control Center, we accomplished the first participation of PCCs in standards-based HIE in the US. We faced several challenges that are also likely to be present at PCCs in other states, including the lack of a robust set of patient identifiers to support automated patient identity matching, challenges in emergency department computerized workflow integration, and the need to build HIE software for PCCs. CONCLUSION: As a multi-disciplinary, multi-organizational team, we successfully developed both a process and the informatics tools necessary to enable PCC participation in standards-based HIE and implemented the process at the Utah PCC.

November 2020 is the 30th anniversary of the release of the first edition of the Unified Medical Language System (UMLS) Knowledge Sources1 by the US National Library of Medicine (NLM). This special issue of JAMIA celebrates an early milestone in Open Science by highlighting research that makes use of the current versions of the UMLS resources. A previous special section of JAMIA, published in 1998, helped to mark the first decade of UMLS research and development.2 The 1990 UMLS Knowledge Sources3 were the result of a large multisite and multidisciplinary research project launched by NLM Director Donald Lindberg 4 years earlier.4 They were the first iteration of novel and freely distributed digital resources intended to help computers behave as if they understand biomedical meaning. In other words, the UMLS Knowledge Sources were designed to help developers produce systems able to retrieve and integrate semantically related information from disparate electronic sources (eg, literature, electronic health records [EHRs], databanks, research registries), irrespective of significant differences in the vocabularies and code sets used within them. In 1990, the problem the UMLS was targeting was not as obvious as the Web would soon make it, the idea of customizing multi-purpose resources was unfamiliar to informatics developers, the notion of a Semantic Network to represent biomedical “common sense” was strange, and the format of the first version of the UMLS Metathesaurus was difficult to understand. With 64 123 concepts and 208 559 concept names from 7 sources, the 1990 Metathesaurus was both too small (in concept coverage) and too large (in file size) for many informatics groups. As a result, NLM and the institutions funded by the UMLS project were the principal users of the first several versions of the UMLS Knowledge Sources.5 Fortunately, early use was sufficient to provide important feedback. Although complexity remains an issue, the free and regularly updated, expanded, and enhanced UMLS Knowledge Sources are a good illustration of Lindberg’s maxim that “systems that get used get better,” thereby driving yet more use. As a premier example, the 1994 addition of the SPECIALIST Lexicon and Lexical Tools,6 and their use to index the Metathesaurus, made the UMLS Knowledge Sources an unparalleled free resource for biomedical natural language processing (NLP) and had a major impact on research in that field. The increasing availability of UMLS-related tools, such as MetamorphoSys,7 MetaMap,8 cTAKES,9 and CLAMP,10 produced at NLM and elsewhere, helped users to cope with customization and complexity and also drove additional use. Perhaps even more important, external developments in information technology,11 enormous increases in genomic and EHR data, and new biomedical research priorities [eg,12] promoted use of the UMLS Knowledge Sources and tools. These developments reduced technical barriers to UMLS use, [eg,13] fostered greater understanding of the UMLS goal, and provided compelling new use cases related to maintenance and use of EHR standards, quality measures, [eg, 14,15] and clinical decision tools; analysis and research use of observational health data; [eg,16,17] and knowledge discovery across heterogeneous data sources [eg,18]. Thirty years after the release of the first experimental edition, the UMLS Knowledge Sources underpin a wide variety of consequential informatics research and applications. The Metathesaurus (2020AA edition) now contains 4.28 million concepts and 15.5 million concept names (including some in 25 different spoken languages) from 214 vocabulary sources; the Semantic Network has 127 types and 54 relationships; the SPECIALIST Lexicon includes 983 420 lexical items; and there are many associated lexical programs and tools. Seen in hindsight, some features that add complexity to the UMLS Metathesaurus also make terminology data more FAIR (findable, accessible, interoperable, and reusable).19 The articles in this Special Issue include examples of just some of the many types of research and development involving the UMLS resources. Two articles provide complementary overviews of UMLS users and uses giving a broader picture of UMLS impact beyond the specific research and applications highlighted in this issue. Drawing on data from annual reports from more than 5000 UMLS users, statistics on download and API accesses, and a scoping review of a random sample of recent research literature, Amos et al present a perspective on the heavy direct use of UMLS resources, including in many production applications and commercial systems not reflected in published literature, and comment on how this use aligns with the stated purpose of the UMLS.20 Kim at al present the results of a scientometric review of more than 10 000 bibliographic records for UMLS related publications from 1986–2019, mapping the multiple disciplines and themes involved in published UMLS work.21 A perennial type of UMLS research is aimed at enhancing the content, characteristics, and distribution formats of the UMLS resources. Two articles in this issue fit this category. A case report by Lu et al describes recent developments in the SPECIALIST Lexicon and lexical tools that further enhance their utility for NLP research and applications.22 Vasilakes et al report the results of an assessment of the coverage of dietary supplement terminology in the UMLS Metathesaurus and identify a method for improving it.23 Key value-added contributions of the UMLS Metathesaurus are concept organization (cross-source synonymy), assignment of semantic types to concepts, and representation of all source vocabularies in a single common machine-readable format. In a number of cases, eg, the International Classification of Diseases, 9th Edition, Clinical Modification, the initial incorporation of a source vocabulary into the Metathesaurus was the first time that its semantics were explicitly represented in machine-readable form. The fully specified Metathesaurus format enabled novel research on terminology auditing techniques that have been applied both to the unique properties of the Metathesaurus and to assessment and comparison of its source vocabularies. Two articles in this issue focus on this line of research. L. Zheng, He, et al review the published literature on auditing techniques applied to the unique semantic features of the UMLS Metathesaurus,24 and F. Zheng et al introduce a transformation-based method for auditing IS-A hierarchies.25 As free and regularly enhanced linguistic and semantic resources, the UMLS Knowledge Sources and related tools promoted interest in biomedical and clinical NLP and have been heavily used in developing, testing, and comparing methods for biomedical information retrieval and extraction tasks. Weinzierl at al explore how knowledge embeddings learned from the UMLS affect the quality of relation extraction from text.26 In work focused on automated measurement of the semantic relatedness among concepts, Mao and Fung describe methods that rely on public off-the-shelf word and graph embedding tools and the UMLS as the sole corpus.27 UMLS resources have been employed in developing test collections for NLP challenges, including some sponsored by the Text REtrieval Conference (TREC), BioCreAtIvE (Critical Assessment of Information Extraction systems in Biology), and the National NLP Clinical Challenges (n2c2, formerly known as i2b2 NLP Shared Tasks), and have been used in systems developed by many challenge participants. Three articles stem from the 2019 n2c2/Open Health NLP shared task on clinical concept normalization, with a goal of mapping terms in clinical text to UMLS concept unique identifiers (CUIs). Henry and Uzuner provide an overview of the shared task and its results, comparing the participating systems.28 Chen at al29 and Xu et al30 each describe a high performing entry in the shared task and enhancements that leverage the UMLS made after the task evaluation. In addition to their importance in research methods, the UMLS Knowledge Sources and tools are essential to the development and maintenance of a very wide range of applications. Four articles in this issue provide examples of the diversity of problems to which UMLS resources are applied. Reimer et al used the UMLS in building the Transport Data Repository to enable study of patients who undergo medical transfer.31 Wang et al determined that use of the UMLS could improve automated categorization of patient safety incident reports.32 Rasmy et al found the UMLS useful in representing electronic health record (EHR) data in predictive modeling.33 Bitton et al mapped transliterated terms to UMLS concepts to improve retrieval in a Hebrew online health community,34 an example of using the UMLS both to extract information from social media and to aid interpretation of non-English text. Thirty years after the initial UMLS release, differences in vocabularies and codes used in different digital information sources—and in the terminology employed by different users—show no signs of disappearing, despite progress in standardizing some data elements in EHRs. [eg,35] The UMLS currently helps thousands of system developers and researchers to overcome variations in the way concepts are expressed, a task that remains critical to effective retrieval, analysis, aggregation, and semantically interoperable exchange of biomedical and health-related information and data. Use of the UMLS resources underpins systems collectively used by millions of scientists, health professionals, patients, and consumers—and by thousands of computer programs—every day. A precise accounting of UMLS use and impact is not possible, however. Statistics and published papers are not available for many commercial products and institution-specific systems that rely on the UMLS. The first UMLS resources were conceived and produced before the arrival of many things that define the informatics landscape today. These include widespread public access to high-speed Internet, Web browsers, search engines that rely on precomputed connections, inexpensive computers and information appliances, immense quantities of digital information and “big data,” and scientific and public policy priorities aimed at leveraging and advancing these developments. For the past 30 years, changes in the broad informatics environment and in biomedical and artificial intelligence research have facilitated the production, expansion, distribution, and use of the UMLS resources. They have also increased, rather than reduced, UMLS utility. It remains to be seen whether this pattern will hold into the long-term future. In the meantime, NLM will serve the informatics field—and users of biomedical and health information everywhere—by continuing to update, enhance, and simplify the UMLS Knowledge Sources, taking advantage of user feedback, ongoing developments in information technology, and advances in research methods that the UMLS itself has helped to foster. BLH retired from the NLM in 2017. She served as UMLS Project Director at NLM from 1986–2006. All authors participated in defining and organizing the topics covered and in drafting and editing the manuscript. All authors approved the final version of the manuscript. This special issue is dedicated to the memory of Donald Allan Bror Lindberg, MD (1933–2019). An informatics pioneer, leader, and longest serving Director of the US National Library of Medicine (NLM), Dr. Lindberg conceived, initiated, and led the Unified Medical Language System (UMLS) effort in its formative years and provided sustained support for free dissemination and regular maintenance of the UMLS resources.

OBJECTIVE: To determine the impact of a graphical information display on diagnosing circulatory shock. MATERIALS AND METHODS: This was an experimental study comparing integrated and conventional information displays. Participants were intensivists or critical care fellows (experts) and first-year medical residents (novices). RESULTS: The integrated display was associated with higher performance (87% vs 82%; P < .001), less time (2.9 vs 3.5 min; P = .008), and more accurate etiology (67% vs 54%; P = .048) compared to the conventional display. When stratified by experience, novice physicians using the integrated display had higher performance (86% vs 69%; P < .001), less time (2.9 vs 3.7 min; P = .03), and more accurate etiology (65% vs 42%; P = .02); expert physicians using the integrated display had nonsignificantly improved performance (87% vs 82%; P = .09), time (2.9 vs 3.3; P = .28), and etiology (69% vs 67%; P = .81). DISCUSSION: The integrated display appeared to support efficient information processing, which resulted in more rapid and accurate circulatory shock diagnosis. Evidence more strongly supported a difference for novices, suggesting that graphical displays may help reduce expert-novice performance gaps.

OBJECTIVE: The study sought to describe key features of clinical concepts and data required to implement clinical practice recommendations as clinical decision support (CDS) tools in electronic health record systems and to identify recommendation features that predict feasibility of implementation. MATERIALS AND METHODS: Using semistructured interviews, CDS implementers and clinician subject matter experts from 7 academic medical centers rated the feasibility of implementing 10 American College of Emergency Physicians Choosing Wisely Recommendations as electronic health record-embedded CDS and estimated the need for additional data collection. Ratings were combined with objective features of the guidelines to develop a predictive model for technical implementation feasibility. RESULTS: A linear mixed model showed that the need for new data collection was predictive of lower implementation feasibility. The number of clinical concepts in each recommendation, need for historical data, and ambiguity of clinical concepts were not predictive of implementation feasibility. CONCLUSIONS: The availability of data and need for additional data collection are essential to assess the feasibility of CDS implementation. Authors of practice recommendations and guidelines can enable organizations to more rapidly assess data availability and feasibility of implementation by including operational definitions for required data.

OBJECTIVE: The study sought to identify barriers to and facilitators of point-of-care information seeking and use of knowledge resources. MATERIALS AND METHODS: We searched MEDLINE, Embase, PsycINFO, and Cochrane Library from 1991 to February 2017. We included qualitative studies in any language exploring barriers to and facilitators of point-of-care information seeking or use of electronic knowledge resources. Two authors independently extracted data on users, study design, and study quality. We inductively identified specific barriers or facilitators and from these synthesized a model of key determinants of information-seeking behaviors. RESULTS: Forty-five qualitative studies were included, reporting data derived from interviews (n = 26), focus groups (n = 21), ethnographies (n = 6), logs (n = 4), and usability studies (n = 2). Most studies were performed within the context of general medicine (n = 28) or medical specialties (n = 13). We inductively identified 58 specific barriers and facilitators and then created a model reflecting 5 key determinants of information-seeking behaviors: time includes subthemes of time availability, efficiency of information seeking, and urgency of information need; accessibility includes subthemes of hardware access, hardware speed, hardware portability, information restriction, and cost of resources; personal skills and attitudes includes subthemes of computer literacy, information-seeking skills, and contextual attitudes about information seeking; institutional attitudes, cultures, and policies includes subthemes describing external individual and institutional information-seeking influences; and knowledge resource features includes subthemes describing information-seeking efficiency, information content, information organization, resource familiarity, information credibility, information currency, workflow integration, compatibility of recommendations with local processes, and patient educational support. CONCLUSIONS: Addressing these determinants of information-seeking behaviors may facilitate clinicians' question answering to improve patient care.

OBJECTIVE: Clinician information overload is prevalent in critical care settings. Improved visualization of patient information may help clinicians cope with information overload, increase efficiency, and improve quality. We compared the effect of information display interventions with usual care on patient care outcomes. MATERIALS AND METHODS: We conducted a systematic review including experimental and quasi-experimental studies of information display interventions conducted in critical care and anesthesiology settings. Citations from January 1990 to June 2018 were searched in PubMed and IEEE Xplore. Reviewers worked independently to screen articles, evaluate quality, and abstract primary outcomes and display features. RESULTS: Of 6742 studies identified, 22 studies evaluating 17 information displays met the study inclusion criteria. Information display categories included comprehensive integrated displays (3 displays), multipatient dashboards (7 displays), physiologic and laboratory monitoring (5 displays), and expert systems (2 displays). Significant improvement on primary outcomes over usual care was reported in 12 studies for 9 unique displays. Improvement was found mostly with comprehensive integrated displays (4 of 6 studies) and multipatient dashboards (5 of 7 studies). Only 1 of 5 randomized controlled trials had a positive effect in the primary outcome. CONCLUSION: We found weak evidence suggesting comprehensive integrated displays improve provider efficiency and process outcomes, and multipatient dashboards improve compliance with care protocols and patient outcomes. Randomized controlled trials of physiologic and laboratory monitoring displays did not show improvement in primary outcomes, despite positive results in simulated settings. Important research translation gaps from laboratory to actual critical care settings exist.

Introduction: Many electronic health records fail to support information uptake because they impose low-level information organization tasks on users. Clinical concept-oriented views have shown information processing improvements, but the specifics of this organization for critical care are unclear. Objective: To determine high-level cognitive processes and patient information organization schema in critical care. Methods: We conducted an open card sort of 29 patient data elements and a modified Delphi card sort of 65 patient data elements. Study participants were 39 clinicians with varied critical care training and experience. We analyzed the open sort with a hierarchical cluster analysis (HCA) and factor analysis (FA). The Delphi sort was split into three initiating groups that resulted in three unique solutions. We compared results between open sort analyses (HCA and FA), between card sorting exercises (open and Delphi), and across the Delphi solutions. Results: Between the HCA and FA, we observed common constructs including cardiovascular and hemodynamics, infectious disease, medications, neurology, patient overview, respiratory, and vital signs. The more comprehensive Delphi sort solutions also included gastrointestinal, renal, and imaging constructs. Conclusions: We identified primarily system-based groupings (e.g., cardiovascular, respiratory). Source-based (e.g., medications, laboratory) groups became apparent when participants were asked to sort a longer list of concepts. These results suggest a hybrid approach to information organization, which may combine systems, source, or problem-based groupings, best supports clinicians' mental models. These results can contribute to the design of information displays to better support clinicians' access and interpretation of information for critical care decisions.

OBJECTIVE: Infobuttons appear as small icons adjacent to electronic health record (EHR) data (e.g., medications, diagnoses, or test results) that, when clicked, access online knowledge resources tailored to the patient, care setting, or task. Infobuttons are required for "Meaningful Use" certification of US EHRs. We sought to evaluate infobuttons' impact on clinical practice and identify features associated with improved outcomes. METHODS: We conducted a systematic review, searching MEDLINE, EMBASE, and other databases from inception to July 6, 2015. We included and cataloged all original research in any language describing implementation of infobuttons or other context-sensitive links. Studies evaluating clinical implementations with outcomes of usage or impact were reviewed in greater detail. Reviewers worked in duplicate to select articles, evaluate quality, and abstract information. RESULTS: Of 599 potential articles, 77 described infobutton implementation. The 17 studies meriting detailed review, including 3 randomized trials, yielded the following findings. Infobutton usage frequency ranged from 0.3 to 7.4 uses per month per potential user. Usage appeared to be influenced by EHR task. Five studies found that infobuttons are used less often than non-context-sensitive links (proportionate usage 0.20-0.34). In 3 studies, users answered their clinical question in > 69% of infobutton sessions. Seven studies evaluated alternative approaches to infobutton design and implementation. No studies isolated the impact of infobuttons on objectively measured patient outcomes. CONCLUSIONS: Weak evidence suggests that infobuttons can help providers answer clinical questions. Research on optimal infobutton design and implementation, and on the impact on patient outcomes and provider behaviors, is needed.

OBJECTIVES: Identify and describe information needs and associated goals of physicians, care coordinators, and families related to coordinating care for medically complex children and youth with special health care needs (CYSHCN). MATERIALS AND METHODS: We conducted 19 in-depth interviews with physicians, care coordinators, and parents of CYSHCN following the Critical Decision Method technique. We analyzed the interviews for information needs posed as questions using a systematic content analysis approach and categorized the questions into information need goal types and subtypes. RESULTS: The Critical Decision Method interviews resulted in an average of 80 information needs per interview. We categorized them into 6 information need goal types: (1) situation understanding, (2) care networking, (3) planning, (4) tracking/monitoring, (5) navigating the health care system, and (6) learning, and 32 subtypes. DISCUSSION AND CONCLUSION: Caring for CYSHCN generates a large amount of information needs that require significant effort from physicians, care coordinators, parents, and various other individuals. CYSHCN are often chronically ill and face developmental challenges that translate into intense demands on time, effort, and resources. Care coordination for CYCHSN involves multiple information systems, specialized resources, and complex decision-making. Solutions currently offered by health information technology fall short in providing support to meet the information needs to perform the complex care coordination tasks. Our findings present significant opportunities to improve coordination of care through multifaceted and fully integrated informatics solutions.

OBJECTIVES: The practice of evidence-based medicine involves integrating the latest best available evidence into patient care decisions. Yet, critical barriers exist for clinicians' retrieval of evidence that is relevant for a particular patient from primary sources such as randomized controlled trials and meta-analyses. To help address those barriers, we investigated machine learning algorithms that find clinical studies with high clinical impact from PubMed®. METHODS: Our machine learning algorithms use a variety of features including bibliometric features (e.g., citation count), social media attention, journal impact factors, and citation metadata. The algorithms were developed and evaluated with a gold standard composed of 502 high impact clinical studies that are referenced in 11 clinical evidence-based guidelines on the treatment of various diseases. We tested the following hypotheses: (1) our high impact classifier outperforms a state-of-the-art classifier based on citation metadata and citation terms, and PubMed's® relevance sort algorithm; and (2) the performance of our high impact classifier does not decrease significantly after removing proprietary features such as citation count. RESULTS: The mean top 20 precision of our high impact classifier was 34% versus 11% for the state-of-the-art classifier and 4% for PubMed's® relevance sort (p=0.009); and the performance of our high impact classifier did not decrease significantly after removing proprietary features (mean top 20 precision=34% vs. 36%; p=0.085). CONCLUSION: The high impact classifier, using features such as bibliometrics, social media attention and MEDLINE® metadata, outperformed previous approaches and is a promising alternative to identifying high impact studies for clinical decision support.

OBJECTIVE: Infobuttons are clinical decision tools embedded in the electronic health record that attempt to link clinical data with context sensitive knowledge resources. We systematically reviewed technical approaches that contribute to improved infobutton design, implementation and functionality. METHODS: We searched databases including MEDLINE, EMBASE, and the Cochrane Library database from inception to March 1, 2016 for studies describing the use of infobuttons. We selected full review comparative studies, usability studies, and qualitative studies examining infobutton design and implementation. We abstracted usability measures such as user satisfaction, impact, and efficiency, as well as prediction accuracy of infobutton content retrieval algorithms and infobutton adoption/interoperability. RESULTS: We found 82 original research studies on infobuttons. Twelve studies met criteria for detailed abstraction. These studies investigated infobutton interoperability (1 study); tools to help tailor infobutton functionality (1 study); interventions to improve user experience (7 studies); and interventions to improve content retrieval by improving prediction of relevant knowledge resources and information needs (3 studies). In-depth interviews with implementers showed the Health Level Seven (HL7) Infobutton standard to be simple and easy to implement. A usability study demonstrated the feasibility of a tool to help medical librarians tailor infobutton functionality. User experience studies showed that access to resources with which users are familiar increased user satisfaction ratings; and that links to specific subsections of drug monographs increased information seeking efficiency. However, none of the user experience improvements led to increased usage uptake. Recommender systems based on machine learning algorithms outperformed hand-crafted rules in the prediction of relevant resources and clinicians' information needs in a laboratory setting, but no studies were found using these techniques in clinical settings. Improved content indexing in one study led to improved content retrieval across three health care organizations. CONCLUSION: Best practice technical approaches to ensure optimal infobutton functionality, design and implementation remain understudied. The HL7 Infobutton standard has supported wide adoption of infobutton functionality among clinical information systems and knowledge resources. Limited evidence supports infobutton enhancements such as links to specific subtopics, configuration of optimal resources for specific tasks and users, and improved indexing and content coverage. Further research is needed to investigate user experience improvements to increase infobutton use and effectiveness.

OBJECTIVES: Data extraction from original study reports is a time-consuming, error-prone process in systematic review development. Information extraction (IE) systems have the potential to assist humans in the extraction task, however majority of IE systems were not designed to work on Portable Document Format (PDF) document, an important and common extraction source for systematic review. In a PDF document, narrative content is often mixed with publication metadata or semi-structured text, which add challenges to the underlining natural language processing algorithm. Our goal is to categorize PDF texts for strategic use by IE systems. METHODS: We used an open-source tool to extract raw texts from a PDF document and developed a text classification algorithm that follows a multi-pass sieve framework to automatically classify PDF text snippets (for brevity, texts) into TITLE, ABSTRACT, BODYTEXT, SEMISTRUCTURE, and METADATA categories. To validate the algorithm, we developed a gold standard of PDF reports that were included in the development of previous systematic reviews by the Cochrane Collaboration. In a two-step procedure, we evaluated (1) classification performance, and compared it with machine learning classifier, and (2) the effects of the algorithm on an IE system that extracts clinical outcome mentions. RESULTS: The multi-pass sieve algorithm achieved an accuracy of 92.6%, which was 9.7% (p<0.001) higher than the best performing machine learning classifier that used a logistic regression algorithm. F-measure improvements were observed in the classification of TITLE (+15.6%), ABSTRACT (+54.2%), BODYTEXT (+3.7%), SEMISTRUCTURE (+34%), and MEDADATA (+14.2%). In addition, use of the algorithm to filter semi-structured texts and publication metadata improved performance of the outcome extraction system (F-measure +4.1%, p=0.002). It also reduced of number of sentences to be processed by 44.9% (p<0.001), which corresponds to a processing time reduction of 50% (p=0.005). CONCLUSIONS: The rule-based multi-pass sieve framework can be used effectively in categorizing texts extracted from PDF documents. Text classification is an important prerequisite step to leverage information extraction from PDF documents.

OBJECTIVE: Poison control centers (PCCs) routinely collaborate with emergency departments (EDs) to provide care for poison-exposed patients. During this process, a significant amount of information is exchanged between EDs and PCCs via telephone, leading to important inefficiencies and safety vulnerabilities. In the present work, we identified and assessed a set of data standards to enable a standards-based health information exchange process between EDs and PCCs. MATERIALS AND METHODS: Based on a reference model for PCC-ED health information exchange, we (1) mapped PCC-ED information exchange events to clinical documents specified in the Health Level Seven (HL7) Consolidated Clinical Document Architecture (C-CDA) Standard, and (2) mapped information types routinely exchanged in PCC-ED telephone conversations to C-CDA sections. RESULTS: Four C-CDA document types were necessary to support the PCC-ED information exchange process: History & Physical Note, Consultation Note, Progress Note, and Discharge Summary. Information types that are commonly exchanged between PCCs and EDs can be reasonably well represented within these C-CDA documents. CONCLUSIONS: A standards-based health information exchange process between PCCs and EDs appears to be feasible given a set of clinical data standards that are required for EHR certification in the USA, although the proposed approach still needs to be validated in actual system implementations. Such a process has the potential to improve the safety and efficiency of PCC-ED communication, ultimately resulting in improved patient care outcomes.

OBJECTIVE: Literature database search is a crucial step in the development of clinical practice guidelines and systematic reviews. In the age of information technology, the process of literature search is still conducted manually, therefore it is costly, slow and subject to human errors. In this research, we sought to improve the traditional search approach using innovative query expansion and citation ranking approaches. METHODS: We developed a citation retrieval system composed of query expansion and citation ranking methods. The methods are unsupervised and easily integrated over the PubMed search engine. To validate the system, we developed a gold standard consisting of citations that were systematically searched and screened to support the development of cardiovascular clinical practice guidelines. The expansion and ranking methods were evaluated separately and compared with baseline approaches. RESULTS: Compared with the baseline PubMed expansion, the query expansion algorithm improved recall (80.2% vs. 51.5%) with small loss on precision (0.4% vs. 0.6%). The algorithm could find all citations used to support a larger number of guideline recommendations than the baseline approach (64.5% vs. 37.2%, p<0.001). In addition, the citation ranking approach performed better than PubMed's "most recent" ranking (average precision +6.5%, recall@k +21.1%, p<0.001), PubMed's rank by "relevance" (average precision +6.1%, recall@k +14.8%, p<0.001), and the machine learning classifier that identifies scientifically sound studies from MEDLINE citations (average precision +4.9%, recall@k +4.2%, p<0.001). CONCLUSIONS: Our unsupervised query expansion and ranking techniques are more flexible and effective than PubMed's default search engine behavior and the machine learning classifier. Automated citation finding is promising to augment the traditional literature search.

OBJECTIVE: The amount of information for clinicians and clinical researchers is growing exponentially. Text summarization reduces information as an attempt to enable users to find and understand relevant source texts more quickly and effortlessly. In recent years, substantial research has been conducted to develop and evaluate various summarization techniques in the biomedical domain. The goal of this study was to systematically review recent published research on summarization of textual documents in the biomedical domain. MATERIALS AND METHODS: MEDLINE (2000 to October 2013), IEEE Digital Library, and the ACM digital library were searched. Investigators independently screened and abstracted studies that examined text summarization techniques in the biomedical domain. Information is derived from selected articles on five dimensions: input, purpose, output, method and evaluation. RESULTS: Of 10,786 studies retrieved, 34 (0.3%) met the inclusion criteria. Natural language processing (17; 50%) and a hybrid technique comprising of statistical, Natural language processing and machine learning (15; 44%) were the most common summarization approaches. Most studies (28; 82%) conducted an intrinsic evaluation. DISCUSSION: This is the first systematic review of text summarization in the biomedical domain. The study identified research gaps and provides recommendations for guiding future research on biomedical text summarization. CONCLUSION: Recent research has focused on a hybrid technique comprising statistical, language processing and machine learning techniques. Further research is needed on the application and evaluation of text summarization in real research or patient care settings.

OBJECTIVE: Online health knowledge resources contain answers to most of the information needs raised by clinicians in the course of care. However, significant barriers limit the use of these resources for decision-making, especially clinicians' lack of time. In this study we assessed the feasibility of automatically generating knowledge summaries for a particular clinical topic composed of relevant sentences extracted from Medline citations. METHODS: The proposed approach combines information retrieval and semantic information extraction techniques to identify relevant sentences from Medline abstracts. We assessed this approach in two case studies on the treatment alternatives for depression and Alzheimer's disease. RESULTS: A total of 515 of 564 (91.3%) sentences retrieved in the two case studies were relevant to the topic of interest. About one-third of the relevant sentences described factual knowledge or a study conclusion that can be used for supporting information needs at the point of care. CONCLUSIONS: The high rate of relevant sentences is desirable, given that clinicians' lack of time is one of the main barriers to using knowledge resources at the point of care. Sentence rank was not significantly associated with relevancy, possibly due to most sentences being highly relevant. Sentences located closer to the end of the abstract and sentences with treatment and comparative predications were likely to be conclusive sentences. Our proposed technical approach to helping clinicians meet their information needs is promising. The approach can be extended for other knowledge resources and information need types.

Point-of-care information needs are common and frequently unmet. One solution to this problem is the use of Infobuttons, which are context-sensitive links from electronic health records (EHR) to knowledge resources, sometimes involving an intermediate broker known as an Infobutton Manager. Health Level Seven (HL7) has developed the Context-Aware Knowledge Retrieval (Infobutton) standard to standardize the integration between EHR systems and knowledge resources. While the standard specifies a set of context attributes and standard terminologies, it leaves to knowledge resources the flexibility to decide how to use these attributes and terminologies to retrieve the most relevant content. This paper describes some of the challenges faced by knowledge resources in trying to locate the most relevant content based on the attribute values for a given Infobutton request. Various approaches to content retrieval are discussed, including the role of indexing with standardized codes, the role of text-based search engines together with their ranking algorithms, and the role of hybrid approaches. Knowledge resource developers must carefully consider business rules, heuristics, and precision/recall tradeoffs when implementing the HL7 Infobutton standard.

OBJECTIVE: Infobuttons are decision support tools that provide links within electronic medical record systems to relevant content in online information resources. The aim of infobuttons is to help clinicians promptly meet their information needs. The objective of this study was to determine whether infobutton links that direct to specific content topics ("topic links") are more effective than links that point to general overview content ("nonspecific links"). DESIGN: Randomized controlled trial with a control and an intervention group. Clinicians in the control group had access to nonspecific links, while those in the intervention group had access to topic links. MEASUREMENTS: Infobutton session duration, number of infobutton sessions, session success rate, and the self-reported impact that the infobutton session produced on decision making. RESULTS: The analysis was performed on 90 subjects and 3,729 infobutton sessions. Subjects in the intervention group spent 17.4% less time seeking for information (35.5 seconds vs. 43 seconds, p = 0.008) than those in the control group. Subjects in the intervention group used infobuttons 20.5% (22 sessions vs. 17.5 sessions, p = 0.21) more often than in the control group, but the difference was not significant. The information seeking success rate was equally high in both groups (89.4% control vs. 87.2% intervention, p = 0.99). Subjects reported a high positive clinical impact (i.e., decision enhancement or knowledge update) in 62% of the sessions. Limitations The exclusion of users with a low frequency of infobutton use and the focus on medication-related information needs may limit the generalization of the results. The session outcomes measurement was based on clinicians' self-assessment and therefore prone to bias. CONCLUSION: The results support the hypothesis that topic links are more efficient than nonspecific links regarding the time seeking for information. It is unclear whether the statistical difference demonstrated will result in a clinically significant impact. However, the overall results confirm previous evidence that infobuttons are effective at helping clinicians to answer questions at the point of care and demonstrate a modest incremental change in the efficiency of information delivery for routine users of this tool.

This paper describes how Intermountain Healthcare's clinical knowledge repository (CKR) uses asset versioning, asset status, and asset linking to manage clinical assets. This allows authors to create, distribute, and maintain complex clinical assets while insuring patient safety. The CKR helps authors to link to existing assets safely and consistently

We describe and report preliminary results of a prototype XML-based method that facilitates the retrieval and navigation of common practice guidelines by physicians at the point of care. The method can be invoked by clicking at "infobuttons" linked to problems in an electronic medical record. Each infobutton displays a list of questions that are categorized and sorted according to the classification proposed by Ely et al. The navigation is achieved through hyperlinks from each question to relevant parts of the guideline. Preliminary results indicate high physician acceptance. A prospective evaluation is now being launched, with the expectation that it will confirm this method as an efficient option for retrieving information from reference documents.

As part of an enterprise effort to develop new clinical information systems at Intermountain Health Care, the authors have built a knowledge authoring tool that facilitates the development and refinement of medical knowledge content. At present, users of the application can compose order sets and an assortment of other structured clinical knowledge documents based on XML schemas. The flexible nature of the application allows the immediate authoring of new types of documents once an appropriate XML schema and accompanying Web form have been developed and stored in a shared repository. The need for a knowledge acquisition tool stems largely from the desire for medical practitioners to be able to write their own content for use within clinical applications. We hypothesize that medical knowledge content for clinical use can be successfully created and maintained through XML-based document frameworks containing structured and coded knowledge.

Medication errors are significant and well-known problems in health care. Despite the evidence supporting the use of computerized physician order entering (CPOE) to help reduce medication errors, only a small number of hospitals in the U.S. have successfully implemented a CPOE system. Different authors have indicated that the utilization of order sets derived from best-practice standards can reduce medication errors and improve physicians' acceptance of CPOE systems. However, a variety of issues related to the development and continuous maintenance of best-practice order sets still need to be understood. This paper presents a model that supports an order set development process driven by clinical experts. Model requirements and details are presented and discussed.

Adverse drug events are known to be a major health problem worldwide. Decision support systems (DSSs) that assist drug ordering have demonstrated to be a powerful tool to prevent prescription errors and adverse drug events. On the other hand, some issues related to the development, implementation, configuration and evaluation of these DDSs still need further research. The objective of this project was the development of a DSS prototype that helps with the prevention of adverse drug events by detecting drug-drug interactions in drug orders. The structure of the system tries to solve some of the problems described by the literature, such as integration with hospital information systems, adaptability to local needs, and knowledge base maintenance. The proposed model has been shown to be an effective method for representing drug-drug interactions.