Systematic reviews (SRs) are vital for evidencebased practice in high stakes disciplines, such as healthcare, but are often impeded by laborintensive and lengthy processes that can span months.Due to the high demand for domain expertise, existing automatic summarization methods fail to accurately identify relevant studies and generate high-quality summaries.To that end, we introduce InsightAgent, a human-centered interactive AI agent powered by large language models that revolutionizes the systematic review workflow.In-sightAgent partitions a large literature corpus based on semantics and employs a multi-agent design for more focused processing of literature, leading to significant improvement in the quality of generated SRs.InsightAgent also provides intuitive visualizations of the corpus and agent trajectories, allowing users to effortlessly monitor the actions of the agent and provide real-time feedback based on their expertise.Our user studies with 9 medical professionals demonstrate that the visualization and interaction mechanisms can effectively improve the quality of synthesized SRs by 27.2%, reaching 79.7% of human-written quality.At the same time, user satisfaction is improved by 34.4%.With InsightAgent, it only takes a clinician about 1.5 hours, rather than months, to complete a high-quality systematic review.InsightAgent demonstrates great potential in facilitating more timely and informed decisionmaking in high stake application scenarios 1 .

OBJECTIVES: Clinicians retrieve data from electronic health record (EHR) systems and summarize them into clinical information to accomplish clinical reasoning and decision-making tasks. Visualization, using meaningful summarization methods and intuitive presentation approaches, can enhance this process. This systematic review examines how EHR data are summarized, visualized, and aligned with the 7 clinical reasoning and decision-making tasks shared by clinicians. MATERIALS AND METHODS: We searched 7 databases for research articles on individual patient EHR related to visualization, clinical decision-support, and patient summaries. Evidence from included studies was extracted for EHR data types, information summarization methods, visualization strategies, clinician characteristics, and evaluations. The synthesized evidence generated data-information-visualization (data-info-vis) flows. RESULTS: We included 112 studies of which 70 (62.5%) conducted detailed usability evaluations, while 42 (37.5%) did not report any evaluations. Gaps remain in deriving actionable insights from EHR data, particularly for tasks requiring data quality reports. Three representative data-info-vis flows emerge. The first uses structured data to generate patterns for temporal visualizations, supporting tasks such as diagnosis and patient management. The second abstracts data into miniature charts, aiding situation-aware understanding and knowledge synthesis. The third features high-level visual metaphors for complex and overarching tasks, such as achieving better care. DISCUSSION AND CONCLUSION: This review identifies 2 primary visualization strategies: (1) timeline-based presentations emphasizing temporal trends and longitudinal tracking, and (2) snapshot-based approaches focusing on status overviews and rapid assessments. The identified critical design approaches and distinct data-info-vis flows are tailored to clinical reasoning and decision-making tasks, offering insights for developing visualization-based decision-support tools.

OBJECTIVES: Health-care institutions customize electronic health record (EHR) configurations to reflect their unique workflows and patient care priorities. Ensuring EHR alignment across sites facilitates seamless information exchange. We developed a pipeline for EHR flowsheet alignment between health-care organizations. The pipeline is augmented by mapping flowsheet data fields to concepts in the Clinical Care Classification (CCC) nursing terminology. MATERIALS AND METHODS: Flowsheet templates and measures from 2 study sites were transformed into template-measure (T-M) pairs. They were aligned through exact, lexical, or semantic matching. Lexical matches were assessed using Jaccard similarity and fuzzy matching methods. Semantic alignment was determined using cosine similarity between large language model-generated embeddings of T-M pairs and CCC concepts to rank and recommend the top n concepts in CCC. Concept mappings were evaluated based on whether concepts were mapped consistently within the CCC hierarchy. RESULTS: We totally aligned 31 255 unique T-M pairs in acute care units and 27 012 T-M pairs in intensive care units from 2 study sites. When restricted to the top-ranked CCC concept (n = 1), we achieved a 63% flowsheet alignment rate with a 53% concept mapping rate. Expanding to the top 3 concepts (n = 3) improved alignment to 96.5% and concept mapping to 96%. DISCUSSION AND CONCLUSION: Electronic health record data field alignment with concept mapping offers opportunities to standardize data elements presented in flowsheets across health-care sites. We demonstrated the feasibility of leveraging a semi-automated pipeline to streamline the EHR flowsheet alignment and accelerate the manual concept mapping process.

In the biomedical domain, visualizing the document embeddings of an extensive corpus has been widely used in information-seeking tasks. However, three key challenges with existing visualizations make it difficult for clinicians to find information efficiently. First, the document embeddings used in these visualizations are generated statically by pretrained language models, which cannot adapt to the user's evolving interest. Second, existing document visualization techniques cannot effectively display how the documents are relevant to users' interest, making it difficult for users to identify the most pertinent information. Third, existing embedding generation and visualization processes suffer from a lack of interpretability, making it difficult to understand, trust and use the result for decision-making. In this paper, we present a novel visual analytics pipeline for user-driven document representation and iterative information seeking (VADIS). VADIS introduces a prompt-based attention model (PAM) that generates dynamic document embedding and document relevance adjusted to the user's query. To effectively visualize these two pieces of information, we design a new document map that leverages a circular grid layout to display documents based on both their relevance to the query and the semantic similarity. Additionally, to improve the interpretability, we introduce a corpus-level attention visualization method to improve the user's understanding of the model focus and to enable the users to identify potential oversight. This visualization, in turn, empowers users to refine, update and introduce new queries, thereby facilitating a dynamic and iterative information-seeking experience. We evaluated VADIS quantitatively and qualitatively on a real-world dataset of biomedical research papers to demonstrate its effectiveness.

A systematic review (SR) is essential with up-to-date research evidence to support clinical decisions and practices. However, the growing literature volume makes it challenging for SR reviewers and clinicians to discover useful information efficiently. Many human-in-the-loop information retrieval approaches (HIR) have been proposed to rank documents semantically similar to users' queries and provide interactive visualizations to facilitate document retrieval. Given that the queries are mainly composed of keywords and keyphrases retrieving documents that are semantically similar to a query does not necessarily respond to the clinician's need. Clinicians still have to review many documents to find the solution. The problem motivates us to develop a visual analytics system, DocFlow, to facilitate information-seeking. One of the features of our DocFlow is accepting natural language questions. The detailed description enables retrieving documents that can answer users' questions. Additionally, clinicians often categorize documents based on their backgrounds and with different purposes (e.g., populations, treatments). Since the criteria are unknown and cannot be pre-defined in advance, existing methods can only achieve categorization by considering the entire information in documents. In contrast, by locating answers in each document, our DocFlow can intelligently categorize documents based on users' questions. The second feature of our DocFlow is a flexible interface where users can arrange a sequence of questions to customize their rules for document retrieval and categorization. The two features of this visual analytics system support a flexible information-seeking process. The case studies and the feedback from domain experts demonstrate the usefulness and effectiveness of our DocFlow.

Many Information Retrieval (IR) approaches have been proposed to extract relevant information from a large corpus. Among these methods, phrase-based retrieval methods have been proven to capture more concrete and concise information than word-based and paragraph-based methods. However, due to the complex relationship among phrases and a lack of proper visual guidance, achieving user-driven interactive information-seeking and retrieval remains challenging. In this study, we present a visual analytic approach for users to seek information from an extensive collection of documents efficiently. The main component of our approach is a PhraseMap, where nodes and edges represent the extracted keyphrases and their relationships, respectively, from a large corpus. To build the PhraseMap, we extract keyphrases from each document and link the phrases according to word attention determined using modern language models, i.e., BERT. As can be imagined, the graph is complex due to the extensive volume of information and the massive amount of relationships. Therefore, we develop a navigation algorithm to facilitate information seeking. It includes (1) a question-answering (QA) model to identify phrases related to users' queries and (2) updating relevant phrases based on users' feedback. To better present the PhraseMap, we introduce a resource-controlled self-organizing map (RC-SOM) to evenly and regularly display phrases on grid cells while expecting phrases with similar semantics to stay close in the visualization. To evaluate our approach, we conducted case studies with three domain experts in diverse literature. The results and feedback demonstrate its effectiveness, usability, and intelligence.

Neural attention-based encoders, which effectively attend sentence tokens to their associated context without being restricted by long-term distance or dependency, have demonstrated outstanding performance in embedding sentences into meaningful representations (embeddings). The Universal Sentence Encoder (USE) is one of the most well-recognized deep neural network (DNN) based solutions, which is facilitated with an attention-driven transformer architecture and has been pre-trained on a large number of sentences from the Internet. Besides the fact that USE has been widely used in many downstream applications, including information retrieval (IR), interpreting its complicated internal working mechanism remains challenging. In this work, we present a visual analytics solution towards addressing this challenge. Specifically, focused on semantics and syntactics (concepts and relations) that are critical to domain clinical IR, we designed and developed a visual analytics system, i.e., USEVis. The system investigates the power of USE in effectively extracting sentences’ semantics and syntactics through exploring and interpreting how linguistic properties are captured by attentions. Furthermore, by thoroughly examining and comparing the inherent patterns of these attentions, we are able to exploit attentions to retrieve sentences/documents that have similar semantics or are closely related to a given clinical problem in IR. By collaborating with domain experts, we demonstrate use cases with inspiring findings to validate the contribution of our work and the effectiveness of our system.

Neural embeddings are widely used in language modeling and feature generation with superior computational power. Particularly, neural document embedding - converting texts of variable-length to semantic vector representations - has shown to benefit widespread downstream applications, e.g., information retrieval (IR). However, the black-box nature makes it difficult to understand how the semantics are encoded and employed. We propose visual exploration of neural document embedding to gain insights into the underlying embedding space, and promote the utilization in prevalent IR applications. In this study, we take an IR application-driven view, which is further motivated by biomedical IR in healthcare decision-making, and collaborate with domain experts to design and develop a visual analytics system. This system visualizes neural document embeddings as a configurable document map and enables guidance and reasoning; facilitates to explore the neural embedding space and identify salient neural dimensions (semantic features) per task and domain interest; and supports advisable feature selection (semantic analysis) along with instant visual feedback to promote IR performance. We demonstrate the usefulness and effectiveness of this system and present inspiring findings in use cases. This work will help designers/developers of downstream applications gain insights and confidence in neural document embedding, and exploit that to achieve more favorable performance in application domains.

Objective: To provide recommendations on how to most effectively implement advanced features of acute care patient portals, including: (1) patient-provider communication, (2) care plan information, (3) clinical data viewing, (4) patient education, (5) patient safety, (6) caregiver access, and (7) hospital amenities. Recommendations: We summarize the experiences of 6 organizations that have implemented acute care portals, representing a variety of settings and technologies. We discuss the considerations for and challenges of incorporating various features into an acute care patient portal, and extract the lessons learned from each institution's experience. We recommend that stakeholders in acute care patient portals should: (1) consider the benefits and challenges of generic and structured electronic care team messaging; (2) examine strategies to provide rich care plan information, such as daily schedule, problem list, care goals, discharge criteria, and post-hospitalization care plan; (3) offer increasingly comprehensive access to clinical data and medical record information; (4) develop alternative strategies for patient education that go beyond infobuttons; (5) focus on improving patient safety through explicit safety-oriented features; (6) consider strategies to engage patient caregivers through portals while remaining cognizant of potential Health Insurance Portability and Accountability Act (HIPAA) violations; (7) consider offering amenities to patients through acute care portals, such as information about navigating the hospital or electronic food ordering.

Time motion studies were first described in the early 20th century in industrial engineering, referring to a quantitative data collection method where an external observer captured detailed data on the duration and movements required to accomplish a specific task, coupled with an analysis focused on improving efficiency. Since then, they have been broadly adopted by biomedical researchers and have become a focus of attention due to the current interest in clinical workflow related factors. However, attempts to aggregate results from these studies have been difficult, resulting from a significant variability in the implementation and reporting of methods. While efforts have been made to standardize the reporting of such data and findings, a lack of common understanding on what "time motion studies" are remains, which not only hinders reviews, but could also partially explain the methodological variability in the domain literature (duration of the observations, number of tasks, multitasking, training rigor and reliability assessments) caused by an attempt to cluster dissimilar sub-techniques. A crucial milestone towards the standardization and validation of time motion studies corresponds to a common understanding, accompanied by a proper recognition of the distinct techniques it encompasses. Towards this goal, we conducted a review of the literature aiming at identifying what is being referred to as "time motion studies". We provide a detailed description of the distinct methods used in articles referenced or classified as "time motion studies", and conclude that currently it is used not only to define the original technique, but also to describe a broad spectrum of studies whose only common factor is the capture and/or analysis of the duration of one or more events. To maintain alignment with the existing broad scope of the term, we propose a disambiguation approach by preserving the expanded conception, while recommending the use of a specific qualifier "continuous observation time motion studies" to refer to variations of the original method (the use of an external observer recording data continuously). In addition, we present a more granular naming for sub-techniques within continuous observation time motion studies, expecting to reduce the methodological variability within each sub-technique and facilitate future results aggregation.

Usability factors are a major obstacle to health information technology (IT) adoption. The purpose of this paper is to review and categorize health IT usability study methods and to provide practical guidance on health IT usability evaluation. 2025 references were initially retrieved from the Medline database from 2003 to 2009 that evaluated health IT used by clinicians. Titles and abstracts were first reviewed for inclusion. Full-text articles were then examined to identify final eligibility studies. 629 studies were categorized into the five stages of an integrated usability specification and evaluation framework that was based on a usability model and the system development life cycle (SDLC)-associated stages of evaluation. Theoretical and methodological aspects of 319 studies were extracted in greater detail and studies that focused on system validation (SDLC stage 2) were not assessed further. The number of studies by stage was: stage 1, task-based or user-task interaction, n=42; stage 2, system-task interaction, n=310; stage 3, user-task-system interaction, n=69; stage 4, user-task-system-environment interaction, n=54; and stage 5, user-task-system-environment interaction in routine use, n=199. The studies applied a variety of quantitative and qualitative approaches. Methodological issues included lack of theoretical framework/model, lack of details regarding qualitative study approaches, single evaluation focus, environmental factors not evaluated in the early stages, and guideline adherence as the primary outcome for decision support system evaluations. Based on the findings, a three-level stratified view of health IT usability evaluation is proposed and methodological guidance is offered based upon the type of interaction that is of primary interest in the evaluation.

OBJECTIVE: To determine whether automated classification of document citations can be useful in reducing the time spent by experts reviewing journal articles for inclusion in updating systematic reviews of drug class efficacy for treatment of disease. DESIGN: A test collection was built using the annotated reference files from 15 systematic drug class reviews. A voting perceptron-based automated citation classification system was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Cross-validation experiments were performed to evaluate performance. MEASUREMENTS: Precision, recall, and F-measure were evaluated at a range of sample weightings. Work saved over sampling at 95% recall was used as the measure of value to the review process. RESULTS: A reduction in the number of articles needing manual review was found for 11 of the 15 drug review topics studied. For three of the topics, the reduction was 50% or greater. CONCLUSION: Automated document citation classification could be a useful tool in maintaining systematic reviews of the efficacy of drug therapy. Further work is needed to refine the classification system and determine the best manner to integrate the system into the production of systematic reviews.