Abstract Patient similarity search is an essential task in healthcare. Recent studies adopted electronic health records (EHRs) to learn patient representations for measuring the clinical similarities. These methods outperformed traditional methods, by capturing more information from various sources consisting of multi-modal EHRs, external knowledge and correlations among medical concepts. They often concerned certain type of data without taking full advantage of various information. We propose a graph representation learning framework, denoted by One-Size-Fits-Three ( OSFT ), that takes into account fusion-attention, neighbor-attention and global-attention from three types of information. Extensive experiments are conducted on two real datasets of MIMIC-III and MIMIC-IV, and the results verified the effectiveness and generality of our framework. When compared with baselines on patient similarity search, our framework achieved good effectiveness and comparative efficiency. The results provide new insights about whether the use of various information can better measure the patient similarity. The source codes are available at https://github.com/emmali808/ADDS/tree/master/EHRDeepHelper .

Medical visual question answering (Med-VQA) is a challenging problem that aims to take a medical image and a clinical question about the image as input and output a correct answer in natural language. Current medical systems often require large-scale and high-quality labeled data for training and evaluation. To address the challenge, we present a new dataset, denoted by OVQA, which is generated from electronic medical records. We develop a semi-automatic data generation tool for constructing the dataset. First, medical entities are automatically extracted from medical records and filled into predefined templates for generating question and answer pairs. These pairs are then combined with medical images extracted from corresponding medical records, to generate candidates for visual question answering (VQA). The candidates are finally verified with high-quality labels annotated by experienced physicians. To evaluate the quality of OVQA, we conduct comprehensive experiments on state-of-the-art methods for the Med-VQA task to our dataset. The results show that our OVQA can be used as a benchmarking dataset for evaluating existing Med-VQA systems. The dataset can be downloaded from http://47.94.174.82/.

This paper demonstrates a medical visual question answering (VQA) system to address three challenges: 1) medical VQA often lacks large-scale labeled training data which requires huge efforts to build; 2) it is costly to implement and thoroughly compare medical VQA models on self-created datasets; 3) applying general VQA models to the medical domain by transfer learning is challenging due to various visual concepts between general images and medical images. Our system has three main components: data generation, model library, and model practice. To address the first challenge, we first allow users to upload self-collected clinical data such as electronic medical records (EMRs) to the data generation component and provides an annotating tool for labeling the data. Then, the system semi-automatically generates medical VQAs for users. Second, we develop a model library by implementing VQA models for users to evaluate their datasets. Users can do simple configurations by selecting self-interested models. The system then automatically trains the models, conducts extensive experimental evaluation, and reports comprehensive findings. The reports provide new insights into the strengths and weaknesses of selected models. Third, we provide an online chat module for users to communicate with an AI robots for further evaluating the models. The source codes are shared on https://github.com/shyanneshan/VQA-Demo.