Health AI suffers from a systemic reproducibility crisis that irreparably hinders research across both academia and industry [4,5].One key tool poised to solve this crisis is the Medical Event Data Standard (MEDS), a comprehensive data format and open-source ecosystem designed to enhance reproducibility and interoperability of AI research using longitudinal Electronic Health Records (EHR) [6].Currently adopted by over 15 institutions globally, MEDS encompasses various open-source tools, published models, and data processing pipelines, enabling streamlined model development and robust benchmarking.In this tutorial, participants will gain key hands-on experience in working with the MEDS format to perform efficient, reproducible, state-of-the-art AI research over real health data.Participants will transform data into the MEDS format, preprocess data, build predictive models, and contribute to the decentralized MEDS-DEV benchmarking platform.Interactive exercises using Jupyter notebooks will provide hands-on experience and practical skills for reproducible health AI research.Attendees will leave equipped

Medical applications of machine learning (ML) have experienced a surge in popularity in recent years. Given the abundance of available data from electronic health records, the intensive care unit (ICU) is a natural habitat for ML. Models have been proposed to address numerous ICU prediction tasks like the early detection of complications. While authors frequently report state-of-the-art performance, it is challenging to verify claims of superiority. Datasets and code are not always published, and cohort definitions, preprocessing pipelines, and training setups are difficult to reproduce. This work introduces Yet Another ICU Benchmark (YAIB), a modular framework that allows researchers to define reproducible and comparable clinical ML experiments; we offer an end-to-end solution from cohort definition to model evaluation. The framework natively supports most open-access ICU datasets (MIMIC III/IV, eICU, HiRID, AUMCdb) and is easily adaptable to future ICU datasets. Combined with a transparent preprocessing pipeline and extensible training code for multiple ML and deep learning models, YAIB enables unified model development, transfer, and evaluation. Our benchmark comes with five predefined established prediction tasks (mortality, acute kidney injury, sepsis, kidney function, and length of stay) developed in collaboration with clinicians. Adding further tasks is straightforward by design. Using YAIB, we demonstrate that the choice of dataset, cohort definition, and preprocessing have a major impact on the prediction performance — often more so than model class — indicating an urgent need for YAIB as a holistic benchmarking tool. We provide our work to the clinical ML community to accelerate method development and enable real-world clinical implementations.

Machine learning (ML) is becoming a core component in query optimizers, e.g., to estimate costs or cardinalities. This means large heterogeneous sets of labeled query plans or jobs (i.e., plans with their runtime or cardinality output) are needed. However, collecting such a training dataset is a very tedious and time-consuming task: It requires both developing numerous jobs and executing them to acquire ground-truth labels. We demonstrate Datafarm,a novel framework for efficiently generating and labeling training data for ML-based query optimizers to overcome these issues. Datafarmenables generating training data tailored to users' needs by learning from their existing workload patterns, input data, and computational resources. It uses an active learning approach to determine a subset of jobs to be executed and encloses the human into the loop, resulting in higher quality data. The graphical user interface of Datafarmallows users to get informative details of the generated jobs and guides them through the generation process step-by-step. We show how users can intervene and provide feedback to the system in an iterative fashion. As an output, users can download both the generated jobs to use as a benchmark and the training data (jobs with their labels).