class MetricDataset(Dataset): def __len__(self): return length Machine Learning (ML) applications are increasingly applied in the healthcare domain, raising the need for so-called trustworthy AI. One fundamental pillar of trustworthy AI is systematic data quality assessment. However, existing ML data quality (DQ) evaluation tools are typically limited to tabular data, lack extensibility and assess only a fraction of data quality aspects. Moreover, many existing tools are unaware of the specific requirements of the given ML task. This prevents them from offering a comprehensive DQ evaluation. To address these issues, we present MetricLib: an extensible toolkit for holistic data quality evaluation of medical ML datasets, based on the theoretical METRIC-framework for trustworthy AI in medicine. The toolkit is able to process a range of data modalities in a memory-efficient manner. While a core set of DQ metrics is implemented, MetricLib is easily extensible with custom metrics and therefore allows investigation of use-case-specific requirements. Additionally, by aggregated DQ scores, the tool enables the efficient identification of data quality gaps. For displaying all results through a graphical user interface, MetricLib is complemented by MetricLibUI. The UI enables targeted, fit-for-purpose data quality analysis based on quantitative and qualitative information. def __getitem__(self, idx): return x, y, metadata User metadata = { "age": 46, "sex": "M", "nurse": "Nurse A", "site": "Site 1", ... } Metric Library Report Creates Metadata MetricLib Dataset PyTorch Dataset Creates User Defined Metric Interface def compute(self, data, reference, metric_config): return { "description": "mean value", "age": data[metric_config["column"]].mean()) } MetricLib Figure 1: Components of Data Quality Evaluation. introduced. It clusters DQ aspects into five categories: measurement process, timeliness, representativeness, informativeness and consistency. Each cluster is further divided into several DQ dimensions and subdimensions. This theoretical framework is equipped with metrics for data quality evaluation [4] and shall now be operationalized in MetricLib. However, a fully automatic, holistic data quality assessment remains intractable, as it requires knowledge of the data source, the target system, the humans involved and particularly the task at hand [10]. Existing Data Quality Tools: A variety of DQ tools already exists [6, 12, 14, 17]. However, these tools are mainly designed to repair errors in tabular data. Additionally, they offer only limited capabilities for analyzing distributional characteristics or complex modalities of medical data, such as imaging or medical time series modalities. Contribution: As a step towards semi-automatic data quality evaluation of medical ML data, we demonstrate MetricLib, an extensible toolkit for modular data quality evaluation. The contribution of MetricLib is twofold: first, MetricLib provides a core set of data quality metrics aligned with the METRIC-framework [16]. Second, the existing data quality metrics can easily be extended with modular, easy-to-use and use-case-specific data quality metrics (by inheriting from an abstract Metric class). Additionally, MetricLib provides data quality scores for the five clusters of the