Motivation: Medical imaging is one of the largest data producers in the world and over the last 30 years this production increased exponentially via a larger number of images and a higher resolution, plus totally new types of images. Most images are used only in the context of a single patient and a single time point, besides a few images that are used for publications or in teaching. Data are usually scattered across many institutions and cannot be combined even for the treatment of a single patient. Much knowledge is stored in these medical archives of images and other clinical information and content-based medical image retrieval has from the start aimed at making such knowledge accessible using visual information in combination with text or structured data. With the digitization of radiology that started in the mid 1990s the foundation for broader use was laid out. Problem statement: This keynote presentation aims at giving a historical perspective of how medical image retrieval has evolved from a few prototypes using first only text, then global visual features to the current multimodal systems that can index many types of images in large quantities and use deep learning as a basis for the tools [1,2,3,4]. It also aims at looking at what the place of image retrieval is in medicine, where it is currently still only sparsely used in clinical practice. It seems that it is mainly a tool for teaching and research. Certified medical tools for decision support rather make use of specific approaches for detection and classification. Approach: The presentation follows a systematic review of the domain that includes many examples of systems and approaches that changed over time when better performing tools became available. Medical mage retrieval has evolved strongly, and many tools linked to mage retrieval are now employed as clinical decision support but mainly for detection and classification. Retrieval remains useful but is often integrated with tools and thus has become almost invisible. A second aspect of the presentation includes a presentations of existing data sets and other resources that were difficult to obtain even ten years ago, but that have been shared via repositories such as TCGA (The Cancer Genome Atlas, https://www.cancer.gov/about-nci/organization/ccg/ research/structural-genomics/tcga), TCIA (The Cancer Imaging Archive, https://www.cancerimagingarchive.net), or via scientific challenges such ImageCLEF [5] or listed in the Grand Challenges web page (https://grand-challenge.org). Medical data are now easily accessible in many fields and often even in large quantities. Discussion: Medical retrieval has gone from single text or image retrieval to multimodal approaches [6], really aiming to use all data available for a case, similar to what a physician would do by looking at a patient holistically. The limiting factor in terms of data access is now rather linked to limited manual annotations, as the time of clinicians for annotations is expensive. Global labels for images usually exist with the associated text reports that describe images and outcomes. Still, these weak labels need to be made usable with deep learning approaches that possibly require large amounts of data to generalize well. Conclusions: Medical image retrieval has evolved strongly over the past 30 years and can be integrated with several tools. For real clinical decision support, it is still rarely used, also because the certification process is tedious and commercial benefit is not as easy to show, as with detection or classification in a clear and limited scenario. In terms of research many resources are available that allow advances also in the future. Still, certification and ethical aspects also need to be taken into account to limit risks for individuals.

Large amounts of medical visual data are produced in hospitals daily and made available continuously via publications in the scientific literature, representing the medical knowledge. However, it is not always easy to find the desired information and in clinical routine the time to fulfil an information need is often very limited. Information retrieval systems are a useful tool to provide access to these documents/images in the biomedical literature related to information needs of medical professionals. Shangri–La is a medical retrieval system that can potentially help clinicians to make decisions on difficult cases. It retrieves articles from the biomedical literature when querying a case description and attached images. The system is based on a multimodal retrieval approach with a focus on the integration of visual information connected to text. The approach includes a query–adaptive multimodal fusion criterion that analyses if visual features are suitable to be fused with text for the retrieval. Furthermore, image modality information is integrated in the retrieval step. The approach is evaluated using the ImageCLEFmed 2013 medical retrieval benchmark and can thus be compared to other approaches. Results show that the final approach outperforms the best multimodal approach submitted to ImageCLEFmed 2013.

No abstract available.

Medical image retrieval can assist physicians in finding information supporting their diagnosis and fulfilling information needs. Systems that allow searching for medical images need to provide tools for quick and easy navigation and query refinement as the time available for information search is often short. Relevance feedback is a powerful tool in information retrieval. This study evaluates relevance feedback techniques with regard to the content they use. A novel relevance feedback technique that uses both text and visual information of the results is proposed. The two information modalities from the image examples are fused either at the feature level using the Rocchio algorithm or at the query list fusion step using a common late fusion rule. Results using the ImageCLEF 2012 benchmark database for medical image retrieval show the potential of relevance feedback techniques in medical image retrieval. The mean average precision (mAP) is used as the evaluation metric and the proposed method outperforms commonly-used methods. The baseline without feedback reached 16 % whereas the relevance feedback with 20 images reached up to 26.35 % with three steps and when using 100 images up to 34.87 % in four steps. Most improvements occur in the first two steps of relevance feedback and then results start to become relatively flat. This might also be due to only using positive feedback as negative feeback often also improves results after more steps. The effect of relevance feedback in automatically spelling corrected and translated queries is investigated as well. Results without mistakes were better than spell-corrected results but the spelling correction more than double results over non-corrected retrieval. Multimodal relevance feedback has shown to be able to help visual medical information retrieval. Next steps include integrating semantics into relevance feedback techniques to benefit from the structured knowledge of ontologies and experimenting on the fusion of text and visual information.

Large amounts of medical images are being produced to help physicians in diagnosis and treatment planning. These images are then archived in PACS (Picture Archival and Communication Systems) and usually they are only reused in the context of the same patient during further visits. Medical image retrieval systems allow medical professionals to search for images in institutional archives, the Internet or in the scientific literature. The goal of the search can be in diagnosis but often as well for teaching and research. A large body of research has investigated efficient and effective algorithms to retrieve a set of images to fulfil a specific information need. However, much less research has been done on studying simple and engaging interaction for users of medical image retrieval systems. In this paper we propose an intuitive and engaging web--based interface targeted to be used by a large range of users with gesture control. This interface allows users to retrieve medical images by accessing a system called Parallel Distributed Image Search Engine (ParaDISE), a text-- and content--based image retrieval system. Accepting search with keywords and example images, this interface uses simple gestures to get random example images and mark examples as positive and negative relevance feedback with results being updated after each interaction.

Medical information is accessible from diverse sources including the general web, social media, journal articles, and hospital records; information searchers can be patients and their families, researchers, practitioners and clinicians. Challenges in medical information retrieval include: diversity of users and user knowledge and expertise; variations in the format, reliability, and quality of biomedical and medical information; the multi-modal nature of much of the data; and the need for accuracy and reliability of medical information. The aim of the workshop is to bring together researchers interested in medical information search with the goal of identifying specific challenges that need to be addressed to advance the state-of-the-art.

Abstract In this article we address the problem of benchmarking image browsers. Image browsers are systems that help the user in finding an image from scratch, as opposed to query by example (QBE), where an example image is needed. The existence of different search paradigms for image browsers makes it difficult to compare image browsers. Currently, the only admissible way of evaluation is by conducting large‐scale user studies. This makes it difficult to use such an evaluation as a tool for improving browsing systems. As a solution, we propose an automatic image browser benchmark that uses structured text annotation of the image collection for the simulation of the user's needs. We apply such a benchmark on an example system.