Problematic opioid use is a major crisis, especially among Veterans of the U.S. Military. In previous work we developed a natural language processing tool to identify problematic opioid use in Veterans Affairs clinical notes. In this work, we developed an application that identifies clinical notes associated with patients who were documented as experiencing problematic opioid use only in clinical notes, as compared to patients who had received a relevant ICD code for problematic opioid use. The application used topic and topic word output from topic modeling to train deep neural network models. The models performed well, achieving area under the curve values of 82% or above, exceeding the performance of baseline models using only topics or topic words. We also computed impact scores, an explainable artificial intelligence method that identifies critical features in training deep learning models. The impact scores extended additional understanding to the data and outcomes.

The field of clinical natural language processing (NLP) has been built on the analysis of clinical sublanguage characteristics. It is well recognized that not only does clinical sublanguage differ from general English (or other languages) but also clinical sublanguage differs among clinical subspecialties and among corpora originating from different healthcare systems. A less recognized aspect is that clinical sublanguage, like all languages, evolves over time. This paper analyses the evolution of clinical sublanguage using a large, national clinical text corpus spanning 15 years. Through the analyses of document types, length, ngrams, and concepts, we found strong evidence that clinical sublanguage does evolve and such changes have implications for NLP development and maintenance. Although the analysis is performed on one corpus, our observations of sublanguage changes are generalizable.

LGBT Patients bear a disproportional burden of health disparities. Sexual orientation and gender identity is of clinical relevance to healthcare providers and data scientists. However, little work has been done to identify LGBT patients, especially in data derived from electronic health record notes. We developed a prototype application that leverages machine learning and rule-based pattern matching methods to identify LGBT patients in a large data source, Veterans Health Administration electronic health record notes. This application achieved 88.2% sensitivity, 91.5% specificity, and 85.9% positive predictive value in a binary classification task for three random document test sets. This work has implications in both improved healthcare and data research.

Irregular spellings in clinical free text present challenges to natural language processing. A number of spelling correction tools exist, but automated spelling correction of clinical text is not a routine practice due to the risk of introducing new errors. We developed a novel spelling analysis application that combines Word2Vec and Levenshtein Edit Distance Constraints to identify variant forms of words. The use case applied to this study was that of discovering herbal and dietary supplements that interact with prescription medications in clinical text. The prototype application processed a large corpus (approximately 1.6 million records), achieving a positive predictive value of 0.9322, in identifying spelling variants, outperforming two baseline methods that achieved positive predictive values of 0.0348 and 0.0067. Our findings suggest that this prototype application provides a more efficient method for researchers and clinicians to find valid misspellings of terms in clinical text.

Sublanguages are domain-centered subsets of general or colloquial language. Their identification drives several language analysis tasks, but it is difficult to discern separate sublanguages in large clinical corpora. We applied k-means clustering of semantic properties, and a novel implementation of relative entropy as an information gain indicator, to identify sublanguages within a large clinical corpus (~1.6 million documents), visualizing the results in a heat map. Patterns both within and across clusters reveal sublanguage trends. These findings are significant in sublanguage analysis, and have implications on both regional and international levels.

Opioid Use Disorder is an international crisis, affecting many populations. Deep learning models can potentially predict opioid use disorder, but provide little insight to how predictions are derived. Impact scores, a new development in explainable artificial intelligence, measure how individual features affect deep learning outcomes. We modeled clinical visits to predict opioid use disorder, computed impact scores, and compared them to odds log ratios from logistic regression. Impact scores were generally comparable to odds log ratios, in providing insight to opioid abuse risk, but from a better-performing method than logistic regression.

Word embedding motivated by deep learning have shown promising results over traditional bag-of-words features for natural language processing. When trained on large text corpora, word embedding methods such as word2vec and doc2vec methods have the advantage of learning from unlabeled data and reduce the dimension of the feature space. In this study, we experimented with word2vec and doc2vec features for a set of clinical text classification tasks and compared the results with using the traditional bag-of-words (BOW) features. The study showed that the word2vec features performed better than the BOW-1-gram features. However, when 2-grams were added to BOW, comparison results were mixed.

Analysis of clinical temporal data can be difficult due to natural properties that often characterize it. The large number of variables, missing values, and other characteristics lead to issues of sparsity and high dimensional complexity. We hypothesized that a pipeline application implementing relevant deep learning methods could sequentially address these difficulties, demonstrating their combined utility in a classification task. We implemented Word2Vec, t-distributed stochastic neighbor embedding, and a convolutional neural network in a pipeline application. To test the pipeline, we applied it to a simple, binary classification task to identify patient encounter care setting. In preliminary testing, the pipeline application achieved 92% accuracy. It also produced temporal data cubes indicative of clinical encounters in intensive care unit (ICU) and non-ICU care settings. A deep learning pipeline process combining multiple methods holds promise in improving analytical tasks of clinical temporal data.

OBJECTIVES: First, to evaluate the effect of standard vs pictograph-enhanced discharge instructions on patients' immediate and delayed recall of and satisfaction with their discharge instructions. Second, to evaluate the effect of automated pictograph enhancement on patient satisfaction with their discharge instructions. MATERIALS AND METHODS: Glyph, an automated healthcare informatics system, was used to automatically enhance patient discharge instructions with pictographs. Glyph was developed at the University of Utah by our research team. Patients in a cardiovascular medical unit were randomized to receive pictograph-enhanced or standard discharge instructions. Measures of immediate and delayed recall and satisfaction with discharge instructions were compared between two randomized groups: pictograph (n = 71) and standard (n = 73). RESULTS: Study participants who received pictograph-enhanced discharge instructions recalled 35% more of their instructions at discharge than those who received standard discharge instructions. The ratio of instructions at discharge was: standard = 0.04 ± 0.03 and pictograph-enhanced = 0.06 ± 0.03. The ratio of instructions at 1 week post discharge was: standard = 0.04 ± 0.02 and pictograph-enhanced 0.04 ± 0.02. Additionally, study participants who received pictograph-enhanced discharge instructions were more satisfied with the understandability of their instructions at 1 week post-discharge than those who received standard discharge instructions. DISCUSSION: Pictograph-enhanced discharge instructions have the potential to increase patient understanding of and satisfaction with discharge instructions. CONCLUSION: It is feasible to automatically illustrate discharge instructions and provide them to patients in a timely manner without interfering with clinical work. Illustrations in discharge instructions were found to improve patients' short-term recall of discharge instructions and delayed satisfaction (1-week post hospitalization) with the instructions. Therefore, it is likely that patients' understanding of and interaction with their discharge instructions is improved by the addition of illustrations.

OBJECTIVE: ClinicalTrials.gov serves critical functions of disseminating trial information to the public and helping the trials recruit participants. This study assessed the readability of trial descriptions at ClinicalTrials.gov using multiple quantitative measures. MATERIALS AND METHODS: The analysis included all 165,988 trials registered at ClinicalTrials.gov as of April 30, 2014. To obtain benchmarks, the authors also analyzed 2 other medical corpora: (1) all 955 Health Topics articles from MedlinePlus and (2) a random sample of 100,000 clinician notes retrieved from an electronic health records system intended for conveying internal communication among medical professionals. The authors characterized each of the corpora using 4 surface metrics, and then applied 5 different scoring algorithms to assess their readability. The authors hypothesized that clinician notes would be most difficult to read, followed by trial descriptions and MedlinePlus Health Topics articles. RESULTS: Trial descriptions have the longest average sentence length (26.1 words) across all corpora; 65% of their words used are not covered by a basic medical English dictionary. In comparison, average sentence length of MedlinePlus Health Topics articles is 61% shorter, vocabulary size is 95% smaller, and dictionary coverage is 46% higher. All 5 scoring algorithms consistently rated CliniclTrials.gov trial descriptions the most difficult corpus to read, even harder than clinician notes. On average, it requires 18 years of education to properly understand these trial descriptions according to the results generated by the readability assessment algorithms. DISCUSSION AND CONCLUSION: Trial descriptions at CliniclTrials.gov are extremely difficult to read. Significant work is warranted to improve their readability in order to achieve CliniclTrials.gov's goal of facilitating information dissemination and subject recruitment.

OBJECTIVES: Natural language processing (NLP) applications typically use regular expressions that have been developed manually by human experts. Our goal is to automate both the creation and utilization of regular expressions in text classification. METHODS: We designed a novel regular expression discovery (RED) algorithm and implemented two text classifiers based on RED. The RED+ALIGN classifier combines RED with an alignment algorithm, and RED+SVM combines RED with a support vector machine (SVM) classifier. Two clinical datasets were used for testing and evaluation: the SMOKE dataset, containing 1091 text snippets describing smoking status; and the PAIN dataset, containing 702 snippets describing pain status. We performed 10-fold cross-validation to calculate accuracy, precision, recall, and F-measure metrics. In the evaluation, an SVM classifier was trained as the control. RESULTS: The two RED classifiers achieved 80.9-83.0% in overall accuracy on the two datasets, which is 1.3-3% higher than SVM's accuracy (p<0.001). Similarly, small but consistent improvements have been observed in precision, recall, and F-measure when RED classifiers are compared with SVM alone. More significantly, RED+ALIGN correctly classified many instances that were misclassified by the SVM classifier (8.1-10.3% of the total instances and 43.8-53.0% of SVM's misclassifications). CONCLUSIONS: Machine-generated regular expressions can be effectively used in clinical text classification. The regular expression-based classifier can be combined with other classifiers, like SVM, to improve classification performance.

OBJECTIVE: We developed a novel computer application called Glyph that automatically converts text to sets of illustrations using natural language processing and computer graphics techniques to provide high quality pictographs for health communication. In this study, we evaluated the ability of the Glyph system to illustrate a set of actual patient instructions, and tested patient recall of the original and Glyph illustrated instructions. METHODS: We used Glyph to illustrate 49 patient instructions representing 10 different discharge templates from the University of Utah Cardiology Service. 84 participants were recruited through convenience sampling. To test the recall of illustrated versus non-illustrated instructions, participants were asked to review and then recall a set questionnaires that contained five pictograph-enhanced and five non-pictograph-enhanced items. RESULTS: The mean score without pictographs was 0.47 (SD 0.23), or 47% recall. With pictographs, this mean score increased to 0.52 (SD 0.22), or 52% recall. In a multivariable mixed effects linear regression model, this 0.05 mean increase was statistically significant (95% CI 0.03 to 0.06, p<0.001). DISCUSSION: In our study, the presence of Glyph pictographs improved discharge instruction recall (p<0.001). Education, age, and English as first language were associated with better instruction recall and transcription. CONCLUSIONS: Automated illustration is a novel approach to improve the comprehension and recall of discharge instructions. Our results showed a statistically significant in recall with automated illustrations. Subjects with no-colleague education and younger subjects appeared to benefit more from the illustrations than others.

OBJECTIVE: This study explores active learning algorithms as a way to reduce the requirements for large training sets in medical text classification tasks. DESIGN: Three existing active learning algorithms (distance-based (DIST), diversity-based (DIV), and a combination of both (CMB)) were used to classify text from five datasets. The performance of these algorithms was compared to that of passive learning on the five datasets. We then conducted a novel investigation of the interaction between dataset characteristics and the performance results. MEASUREMENTS: Classification accuracy and area under receiver operating characteristics (ROC) curves for each algorithm at different sample sizes were generated. The performance of active learning algorithms was compared with that of passive learning using a weighted mean of paired differences. To determine why the performance varies on different datasets, we measured the diversity and uncertainty of each dataset using relative entropy and correlated the results with the performance differences. RESULTS: The DIST and CMB algorithms performed better than passive learning. With a statistical significance level set at 0.05, DIST outperformed passive learning in all five datasets, while CMB was found to be better than passive learning in four datasets. We found strong correlations between the dataset diversity and the DIV performance, as well as the dataset uncertainty and the performance of the DIST algorithm. CONCLUSION: For medical text classification, appropriate active learning algorithms can yield performance comparable to that of passive learning with considerably smaller training sets. In particular, our results suggest that DIV performs better on data with higher diversity and DIST on data with lower uncertainty.

Cohort identification is an important step in conducting clinical research studies. Use of ICD-9 codes to identify disease cohorts is a common approach that can yield satisfactory results in certain conditions; however, for many use-cases more accurate methods are required. In this study, we propose a bootstrapping method that supplements ICD-9 codes with lab results, medications, etc. to build classification models that can be used to identify cohorts more accurately. The proposed method does not require prior information about the true class of the patients. We used the method to identify Diabetes Mellitus (DM) and Hyperlipidemia (HL) patient cohorts from a database of 800 thousand patients. Evaluation results show that the method identified 11,000 patients who did not have DM related ICD-9 codes as positive for DM and 52,000 patients without HL codes as positive for HL. A review of 400 patient charts (200 patients for each condition) by two clinicians shows that in both the conditions studied, the labeling assigned by the proposed approach is more consistent with that of the clinicians compared to labeling through ICD-9 codes. The method is reasonably automated and, we believe, holds potential for inexpensive, more accurate cohort identification.

OBJECTIVE: This study has two objectives: first, to identify and characterize consumer health terms not found in the Unified Medical Language System (UMLS) Metathesaurus (2007 AB); second, to describe the procedure for creating new concepts in the process of building a consumer health vocabulary. How do the unmapped consumer health concepts relate to the existing UMLS concepts? What is the place of these new concepts in professional medical discourse? DESIGN: The consumer health terms were extracted from two large corpora derived in the process of Open Access Collaboratory Consumer Health Vocabulary (OAC CHV) building. Terms that could not be mapped to existing UMLS concepts via machine and manual methods prompted creation of new concepts, which were then ascribed semantic types, related to existing UMLS concepts, and coded according to specified criteria. RESULTS: This approach identified 64 unmapped concepts, 17 of which were labeled as uniquely "lay" and not feasible for inclusion in professional health terminologies. The remaining terms constituted potential candidates for inclusion in professional vocabularies, or could be constructed by post-coordinating existing UMLS terms. The relationship between new and existing concepts differed depending on the corpora from which they were extracted. CONCLUSION: Non-mapping concepts constitute a small proportion of consumer health terms, but a proportion that is likely to affect the process of consumer health vocabulary building. We have identified a novel approach for identifying such concepts.

As the emphasis on individuals' active partnership in health care grows, so does the public's need for effective, comprehensible consumer health resources. Consumer health informatics has the potential to provide frameworks and strategies for designing effective health communication tools that empower users and improve their health decisions. This article presents an overview of the consumer health informatics field, discusses promising approaches to supporting health communication, and identifies challenges plus direction for future research and development. The authors' recommendations emphasize the need for drawing upon communication and social science theories of information behavior, reaching out to consumers via a range of traditional and novel formats, gaining better understanding of the public's health information needs, and developing informatics solutions for tailoring resources to users' needs and competencies. This article was written as a scholarly outreach and leadership project by members of the American Medical Informatics Association's Consumer Health Informatics Working Group.

OBJECTIVES: Effective health communication is often hindered by a "vocabulary gap" between language familiar to consumers and jargon used in medical practice and research. To present health information to consumers in a comprehensible fashion, we need to develop a mechanism to quantify health terms as being more likely or less likely to be understood by typical members of the lay public. Prior research has used approaches including syllable count, easy word list, and frequency count, all of which have significant limitations. DESIGN: In this article, we present a new method that predicts consumer familiarity using contextual information. The method was applied to a large query log data set and validated using results from two previously conducted consumer surveys. MEASUREMENTS: We measured the correlation between the survey result and the context-based prediction, syllable count, frequency count, and log normalized frequency count. RESULTS: The correlation coefficient between the context-based prediction and the survey result was 0.773 (p < 0.001), which was higher than the correlation coefficients between the survey result and the syllable count, frequency count, and log normalized frequency count (p < or = 0.012). CONCLUSIONS: The context-based approach provides a good alternative to the existing term familiarity assessment methods.

OBJECTIVE: Health information retrieval (HIR) on the Internet has become an important practice for millions of people, many of whom have problems forming effective queries. We have developed and evaluated a tool to assist people in health-related query formation. DESIGN: We developed the Health Information Query Assistant (HIQuA) system. The system suggests alternative/additional query terms related to the user's initial query that can be used as building blocks to construct a better, more specific query. The recommended terms are selected according to their semantic distance from the original query, which is calculated on the basis of concept co-occurrences in medical literature and log data as well as semantic relations in medical vocabularies. MEASUREMENTS: An evaluation of the HIQuA system was conducted and a total of 213 subjects participated in the study. The subjects were randomized into 2 groups. One group was given query recommendations and the other was not. Each subject performed HIR for both a predefined and a self-defined task. RESULTS: The study showed that providing HIQuA recommendations resulted in statistically significantly higher rates of successful queries (odds ratio = 1.66, 95% confidence interval = 1.16-2.38), although no statistically significant impact on user satisfaction or the users' ability to accomplish the predefined retrieval task was found. CONCLUSION: Providing semantic-distance-based query recommendations can help consumers with query formation during HIR.

Laypersons ("consumers") often have difficulty finding, understanding, and acting on health information due to gaps in their domain knowledge. Ideally, consumer health vocabularies (CHVs) would reflect the different ways consumers express and think about health topics, helping to bridge this vocabulary gap. However, despite the recent research on mismatches between consumer and professional language (e.g., lexical, semantic, and explanatory), there have been few systematic efforts to develop and evaluate CHVs. This paper presents the point of view that CHV development is practical and necessary for extending research on informatics-based tools to facilitate consumer health information seeking, retrieval, and understanding. In support of the view, we briefly describe a distributed, bottom-up approach for (1) exploring the relationship between common consumer health expressions and professional concepts and (2) developing an open-access, preliminary (draft) "first-generation" CHV. While recognizing the limitations of the approach (e.g., not addressing psychosocial and cultural factors), we suggest that such exploratory research and development will yield insights into the nature of consumer health expressions and assist developers in creating tools and applications to support consumer health information seeking.

OBJECTIVE: There is an abundance of health-related information online, and millions of consumers search for such information. Spell checking is of crucial importance in returning pertinent results, so the authors propose a technique for increasing the effectiveness of spell-checking tools used for health-related information retrieval. DESIGN: A sample of incorrectly spelled medical terms was submitted to two different spell-checking tools, and the resulting suggestions, derived under two different dictionary configurations, were re-sorted according to how frequently each term appeared in log data from a medical search engine. MEASUREMENTS: Univariable analysis was carried out to assess the effect of each factor (spell-checking tool, dictionary type, re-sort, or no re-sort) on the probability of success. The factors that were statistically significant in the univariable analysis were then used in multivariable analysis to evaluate the independent effect of each of the factors. RESULTS: The re-sorted suggestions proved to be significantly more accurate than the original list returned by the spell-checking tool. The odds of finding the correct suggestion in the number one rank were increased by 63% after re-sorting using the authors' method. This effect was independent of both the dictionary and the spell-checking tools that were used. CONCLUSION: Using knowledge about the frequency of a given word's occurrence in the medical domain can significantly improve spelling correction for medical queries.

InterMed is a collaboration among research groups from Stanford, Harvard, and Columbia Universities. The primary goal of InterMed has been to develop a sharable language that could serve as a standard for modeling computer-interpretable guidelines (CIGs). This language, called GuideLine Interchange Format (GLIF), has been developed in a collaborative manner and in an open process that has welcomed input from the larger community. The goals and experiences of the InterMed project and lessons that the authors have learned may contribute to the work of other researchers who are developing medical knowledge-based tools. The lessons described include (1) a work process for multi-institutional research and development that considers different viewpoints, (2) an evolutionary lifecycle process for developing medical knowledge representation formats, (3) the role of cognitive methodology to evaluate and assist in the evolutionary development process, (4) development of an architecture and (5) design principles for sharable medical knowledge representation formats, and (6) a process for standardization of a CIG modeling language.

When a typographic error of a patient identification number occurs on a patient document such as an envelope for radiology films or the cover of a patient record, it will result in misplacement of the document. Once misplaced, such documents are often extremely difficult to recover. After analyzing 290 numerical typos, we found that errors do not occur randomly. Instead, many of the typos share certain specific patterns. Six major types of non-random numeral typographic error patterns have been identified and their frequency characterized. Knowing these patterns and their odds increases the likelihood of finding a misplaced file. In addition, awareness of these patterns during transcribing or writing a patient ID may decrease the chance of typographic errors.

OBJECTIVE: Clinical information systems typically present patient data in chronologic order, organized by the source of the information (e.g., laboratory, radiology). This study evaluates the functionality and utility of a knowledge-based system that generates concept-oriented views (organized around clinical concepts such as disease or organ system) of clinical data. DESIGN: The authors have developed a system that uses a knowledge base of interrelationships between medical concepts to infer relationships between data in electronic medical records. They use these inferences to produce summaries, or views, of the data that are relevant to a specific concept of interest. They evaluated the ability of the system to select relevant information, reduce information overload, and support physician information retrieval. MEASUREMENTS: The sensitivity and specificity of the system for identifying relevant patient information were calculated. Effect on information overload was assessed by comparing the amount of information in each view with the amount of information in the entire record. Information retrieval accuracy and cost (time) were used to measure the effect of using concept-oriented views on the efficiency and effectiveness of retrievals. RESULTS: The sensitivity and specificity of the system for identifying relevant clinical information were generally in the range of 70 to 80 percent. Concept-oriented views are effective in reducing the amount of information retrieved (over 80 percent reduction) and, compared with source-oriented views, are able to improve physician retrieval accuracy (p=0.04). CONCLUSION: Computer-generated, concept-oriented views can be used to reduce clinician information overload and improve the accuracy of clinical data retrieval.