OBJECTIVES: To explore the potential effects of recent restrictions on discussions regarding diversity, equity, and inclusion (DEI) in the field of biomedical informatics. MATERIALS AND METHODS: Executive orders issued by the U.S. federal government regarding diversity and gender issues are discussed in the context of implications for biomedical informatics research. RESULTS: Restrictions on specific terminology can hinder research into critical topics such as bias and fairness in clinical artificial intelligence and machine learning algorithms. Additionally, these limitations may narrow the scope of questions that informatics research can address and obstruct efforts to enhance the diversity of perspectives within the field. DISCUSSION: Responding to these threats requires a community response. The American Medical Informatics Association (AMIA) can help the informatics community present a united front in support of DEI research in multiple ways. CONCLUSION: The informatics community should take a strong and unambiguous response to support diversity, equity, and inclusion of underrepresented perspectives in the field.

MOTIVATION: Software is vital for the advancement of biology and medicine. Impact evaluations of scientific software have primarily emphasized traditional citation metrics of associated papers, despite these metrics inadequately capturing the dynamic picture of impact and despite challenges with improper citation. RESULTS: To understand how software developers evaluate their tools, we conducted a survey of participants in the Informatics Technology for Cancer Research (ITCR) program funded by the National Cancer Institute (NCI). We found that although developers realize the value of more extensive metric collection, they find a lack of funding and time hindering. We also investigated software among this community for how often infrastructure that supports more nontraditional metrics were implemented and how this impacted rates of papers describing usage of the software. We found that infrastructure such as social media presence, more in-depth documentation, the presence of software health metrics, and clear information on how to contact developers seemed to be associated with increased mention rates. Analysing more diverse metrics can enable developers to better understand user engagement, justify continued funding, identify novel use cases, pinpoint improvement areas, and ultimately amplify their software's impact. Challenges are associated, including distorted or misleading metrics, as well as ethical and security concerns. More attention to nuances involved in capturing impact across the spectrum of biomedical software is needed. For funders and developers, we outline guidance based on experience from our community. By considering how we evaluate software, we can empower developers to create tools that more effectively accelerate biological and medical research progress. AVAILABILITY AND IMPLEMENTATION: More information about the analysis, as well as access to data and code is available at https://github.com/fhdsl/ITCR_Metrics_manuscript_website.

Computational models of infectious diseases have become valuable tools for research and the public health response against epidemic threats. The reproducibility of computational models has been limited, undermining the scientific process and possibly trust in modeling results and related response strategies, such as vaccination. We translated published reproducibility guidelines from a wide range of scientific disciplines into an implementation framework for improving reproducibility of infectious disease computational models. The framework comprises 22 elements that should be described, grouped into 6 categories: computational environment, analytical software, model description, model implementation, data, and experimental protocol. The framework can be used by scientific communities to develop actionable tools for sharing computational models in a reproducible way.

Denis Newman-Griffis, Jill Fain Lehman, Carolyn Rosé, Harry Hochheiser. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

OBJECTIVE: This article reports results from a systematic literature review related to the evaluation of data visualizations and visual analytics technologies within the health informatics domain. The review aims to (1) characterize the variety of evaluation methods used within the health informatics community and (2) identify best practices. METHODS: A systematic literature review was conducted following PRISMA guidelines. PubMed searches were conducted in February 2017 using search terms representing key concepts of interest: health care settings, visualization, and evaluation. References were also screened for eligibility. Data were extracted from included studies and analyzed using a PICOS framework: Participants, Interventions, Comparators, Outcomes, and Study Design. RESULTS: After screening, 76 publications met the review criteria. Publications varied across all PICOS dimensions. The most common audience was healthcare providers (n = 43), and the most common data gathering methods were direct observation (n = 30) and surveys (n = 27). About half of the publications focused on static, concentrated views of data with visuals (n = 36). Evaluations were heterogeneous regarding setting and measurements used. DISCUSSION: When evaluating data visualizations and visual analytics technologies, a variety of approaches have been used. Usability measures were used most often in early (prototype) implementations, whereas clinical outcomes were most common in evaluations of operationally-deployed systems. These findings suggest opportunities for both (1) expanding evaluation practices, and (2) innovation with respect to evaluation methods for data visualizations and visual analytics technologies across health settings. CONCLUSION: Evaluation approaches are varied. New studies should adopt commonly reported metrics, context-appropriate study designs, and phased evaluation strategies.

Systematic application of observational data to the understanding of impacts of cancer treatments requires detailed information models allowing meaningful comparisons between treatment regimens. Unfortunately, details of systemic therapies are scarce in registries and data warehouses, primarily due to the complex nature of the protocols and a lack of standardization. Since 2011, we have been creating a curated and semi-structured website of chemotherapy regimens, HemOnc.org. In coordination with the Observational Health Data Sciences and Informatics (OHDSI) Oncology Subgroup, we have transformed a substantial subset of this content into the OMOP common data model, with bindings to multiple external vocabularies, e.g., RxNorm and the National Cancer Institute Thesaurus. Currently, there are >73,000 concepts and >177,000 relationships in the full vocabulary. Content related to the definition and composition of chemotherapy regimens has been released within the ATHENA tool (athena.ohdsi.org) for widespread utilization by the OHDSI membership. Here, we describe the rationale, data model, and initial contents of the HemOnc vocabulary along with several use cases for which it may be valuable.

Cancer is mainly caused by somatic genome alterations (SGAs). Precision oncology involves identifying and targeting tumor-specific aberrations resulting from causative SGAs. We developed a novel tumor-specific computational framework that finds the likely causative SGAs in an individual tumor and estimates their impact on oncogenic processes, which suggests the disease mechanisms that are acting in that tumor. This information can be used to guide precision oncology. We report a tumor-specific causal inference (TCI) framework, which estimates causative SGAs by modeling causal relationships between SGAs and molecular phenotypes (e.g., transcriptomic, proteomic, or metabolomic changes) within an individual tumor. We applied the TCI algorithm to tumors from The Cancer Genome Atlas (TCGA) and estimated for each tumor the SGAs that causally regulate the differentially expressed genes (DEGs) in that tumor. Overall, TCI identified 634 SGAs that are predicted to cause cancer-related DEGs in a significant number of tumors, including most of the previously known drivers and many novel candidate cancer drivers. The inferred causal relationships are statistically robust and biologically sensible, and multiple lines of experimental evidence support the predicted functional impact of both the well-known and the novel candidate drivers that are predicted by TCI. TCI provides a unified framework that integrates multiple types of SGAs and molecular phenotypes to estimate which genome perturbations are causally influencing one or more molecular/cellular phenotypes in an individual tumor. By identifying major candidate drivers and revealing their functional impact in an individual tumor, TCI sheds light on the disease mechanisms of that tumor, which can serve to advance our basic knowledge of cancer biology and to support precision oncology that provides tailored treatment of individual tumors.

The gap between domain experts and natural language processing expertise is a barrier to extracting understanding from clinical text. We describe a prototype tool for interactive review and revision of natural language processing models of binary concepts extracted from clinical notes. We evaluated our prototype in a user study involving 9 physicians, who used our tool to build and revise models for 2 colonoscopy quality variables. We report changes in performance relative to the quantity of feedback. Using initial training sets as small as 10 documents, expert review led to final F1scores for the "appendiceal-orifice" variable between 0.78 and 0.91 (with improvements ranging from 13.26% to 29.90%). F1for "biopsy" ranged between 0.88 and 0.94 (-1.52% to 11.74% improvements). The average System Usability Scale score was 70.56. Subjective feedback also suggests possible design improvements.

OBJECTIVE: To develop and evaluate a pharmacogenomics information resource for pharmacists. MATERIALS AND METHODS: We built a pharmacogenomics information resource presenting Food and Drug Administration (FDA) drug product labelling information, refined it based on feedback from pharmacists, and conducted a comparative usability evaluation, measuring task completion time, task correctness and perceived usability. Tasks involved hypothetical clinical situations requiring interpretation of pharmacogenomics information to determine optimal prescribing for specific patients. RESULTS: Pharmacists were better able to perform certain tasks using the redesigned resource relative to the Pharmacogenomic Knowledgebase (PharmGKB) and the FDA Table of Pharmacogenomic Biomarkers in Drug Labeling. On average, participants completed tasks in 107.5 s using our resource, compared to 188.9 s using PharmGKB and 240.2 s using the FDA table. Using the System Usability Scale, participants rated our resource 79.62 on average, compared to 53.27 for PharmGKB and 50.77 for the FDA table. Participants found the correct answers for 100% of tasks using our resource, compared to 76.9% using PharmGKB and 69.2% using the FDA table. DISCUSSION: We present structured, clinically relevant pharmacogenomic FDA drug product label information with visualizations to help explain the relationships between gene variants, drugs, and phenotypes. The results from our evaluation suggest that user-centered interfaces for pharmacogenomics information can increase ease of access and comprehension. CONCLUSION: A clinician-focused pharmacogenomics information resource can answer pharmacogenomics-related medication questions faster, more correctly, and more easily than widely used alternatives, as perceived by pharmacists.

Computational thinking has been identified as a necessary fundamental skill for all students. University curricula, however, are currently not designed to provide such knowledge to a broad student population. In this paper, we report on our experiences in the development of a model for incorporating computational thinking into the undergraduate, general education curriculum at Towson University. We discuss the model in terms of eliciting faculty interest, institutional support, and positive student response. In the first two years of this NSF-funded three-year project, we have developed, piloted and assessed five computational thinking general education courses - an Everyday Computational Thinking course, and four discipline-specific computational thinking general education courses. Initial assessments show promising and significant student, instructor and administration interest in computational thinking as a basis in courses covering multiple disciplines within the general education curriculum.

Numerous studies have investigated task performance times for selection from hierarchical menus, with structures containing many choices at each of a few levels (broad, shallow structures) generally outperforming structures containing fewer choices at each of many levels (narrow, deep structures). To see if these results applied to blind users who rely on screen reader software for computer access, we replicated an earlier published study, using 19 blind screen-reader users. Consistent with earlier studies, broader, shallow hierarchies outperformed narrow, deep hierarchies. Task performance times and hypertext lostness measures were correlated. Although further work will be needed to understand specific determinants of task performance rates, these results support the use of broad, shallow menus for blind as well as sighted users.

Human-computer interaction (HCI) is much broader than the study of interface design and input devices. It includes considerations of the social, political, ethical, and societal implications of computer systems. Concerns such as privacy, accessibility, universal design, and voting usability have led to active HCI research. Our examination of HCI responses to these and other issues informs a model of social engagement based on societal influences that motivate various responses from the HCI community. This model provides suggestions for engagement with issues that are likely to grow in importance over the next several years. By focusing on these issues, HCI researchers may make still greater contributions toward addressing societal concerns.

Data sets involving linear ordered sequences are a recurring theme in bioinformatics. Dynamic query tools that support exploration of these data sets can be useful for identifying patterns of interest. This paper describes the use of one such tool - timesearcher - to interactively explore linear sequence data sets taken from two bioinformatics problems. Microarray time course data sets involve expression levels for large numbers of genes over multiple time points. Timesearcher can be used to interactively search these data sets for genes with expression profiles of interest. The occurrence frequencies of short sequences of DNA in aligned exons can be used to identify sequences that play a role in the pre-mRNA splicing. Timesearcher can be used to search these data sets for candidate splicing signals.

As a "social protocol" aimed at providing a technological means to address concerns over Internet privacy, the Platform for Privacy Preferences (P3P) has been controversial since its announcement in 1997. In the U.S., critics have decried P3P as an industry attempt to avoid meaningful privacy legislation, while developers have portrayed the proposal as a tool for helping users make informed decisions about the impact of their Web surfing choices. This dispute touches upon the privacy model underlying P3P, the U.S. political context regarding privacy, and the technical components of the protocol. This article presents an examination of these factors, with an eye towards distilling lessons for developers of future social protocols.

The desire to make computing available to broader populations has historically been a motivation for research and innovation that led to new breakthroughs in usability. Menus, graphical user interfaces and the World Wide Web are examples of innovative technological solutions that have arisen out of the challenge of bringing larger and more diverse groups of users into the world of computing. Universal usability is the latest such challenge: In order to build systems that are universally usable, designers must account for technology variety, user diversity and gaps in user knowledge. These issues are particularly challenging and important in the context of increasing the usability of the World Wide Web. To raise awareness, web designers are urged to provide universal usability statements that offer users information about the usability of their sites. These statements can inform users and thereby reduce frustration and confusion. Further steps toward universal usability might be achieved through research aimed at developing tools that would encourage or promote usability. The paper closes with five proposals for future research.

HTTP server log files provide Web site operators with substantial detail regarding the visitors to their sites. Interest in interpreting this data has spawned an active market for software packages that summarize and analyze this data, providing histograms, pie graphs, and other charts summarizing usage patterns. Although useful, these summaries obscure useful information and restrict users to passive interpretation of static displays. Interactive visualizations can be used to provide users with greater abilities to interpret and explore Web log data. By combining two-dimensional displays of thousands of individual access requests, color, and size coding for additional attributes, and facilities for zooming and filtering, these visualizations provide capabilities for examining data that exceed those of traditional Web log analysis tools. We introduce a series of interactive visualizations that can be used to explore server data across various dimensions. Possible uses of these visualizations are discussed, and difficulties of data collection, presentation, and interpretation are explored.

To date, experimental comparisons of menu layouts have concentrated on variants of hierarchical structures of sequentially presented menus. Simultaneous menus-layouts that present multiple active menus on a screen at the same time-are an alternative arrangement that may be useful in many Web design situations. This article describes an experiment involving a between-subject comparison of simultaneous menus and their traditional sequential counterparts. A total of 20 experienced Web users used either simultaneous or sequential menus in a standard Web browser to answer questions based on U.S. Census data. Our results suggest that appropriate use of simultaneous menus can lead to improved task performance speeds without harming subjective satisfaction measures. For novice users performing simple tasks, the simplicity of sequential menus appears to be helpful, but experienced users performing complex tasks may benefit from simultaneous menus. Design improvements can amplify the benefits of simultaneous menu layouts.