MOTIVATION: This work presents the first large-scale neutral benchmark experiment focused on single-event, right-censored, low-dimensional survival data. Benchmark experiments are essential in methodological research to scientifically compare new and existing model classes through proper empirical evaluation. Existing benchmarks in the survival literature are smaller in scale regarding the number of used datasets and extent of empirical evaluation. They often lack appropriate tuning or evaluation procedures, while other comparison studies focus on qualitative reviews rather than quantitative comparisons. This comprehensive study aims to fill the gap by neutrally evaluating a broad range of methods and providing generalizable guidelines for practitioners. RESULTS: We benchmark 21 models, ranging from classical statistical approaches to many common machine learning methods, on 34 publicly available datasets. The benchmark tunes models using both a discrimination measure (Harrell's C-index) and a scoring rule (Integrated Survival Brier Score), and evaluates them across six metrics covering discrimination, calibration, and overall predictive performance. Despite superior average ranks in overall predictive performance from individual learners like oblique random survival forests and likelihood-based boosting, and better discrimination rankings from multiple boosting- and tree-based methods as well as parametric survival models, no method statistically significantly outperforms the commonly used Cox proportional hazards model for either tuning measure. We conclude that while the Cox Proportional Hazards model remains a robust default for low-dimensional, right-censored survival data, more flexible methods may be preferable for specific dataset characteristics. AVAILABILITY AND IMPLEMENTATION: All code, data, and results are publicly available on GitHub https://github.com/slds-lmu/paper_2023_survival_benchmark and archived on Zenodo https://doi.org/10.5281/zenodo.19075310.

Considerable spatial heterogeneity has been observed in COVID-19 transmission across administrative areas of England throughout the pandemic. This study investigates what drives these differences. We constructed a probabilistic case count model for 306 administrative areas of England across 95 weeks, fit using a Bayesian evidence synthesis framework. We incorporate the impact of acquired immunity, of spatial exportation of cases, and 16 spatially-varying socio-economic, socio-demographic, health, and mobility variables. Model comparison assesses the relative contributions of these respective mechanisms. We find that spatially-varying and time-varying differences in week-to-week transmission were definitively associated with differences in: time spent at home, variant-of-concern proportion, and adult social care funding. However, model comparison demonstrates that the impact of these terms is negligible compared to the role of spatial exportation between administrative areas. While these results confirm the impact of some, but not all, static measures of spatially-varying inequity in England, our work corroborates the finding that observed differences in disease transmission during the pandemic were predominantly driven by underlying epidemiological factors rather than aggregated metrics of demography and health inequity between areas. Further work is required to assess how health inequity more broadly contributes to these epidemiological factors.

This paper extends the FAIR (Findable, Accessible, Interoperable, Reusable) guidelines to provide criteria for assessing if software conforms to best practices in open source. By adding "USE" (User-Centered, Sustainable, Equitable), software development can adhere to open source best practice by incorporating user-input early on, ensuring front-end designs are accessible to all possible stakeholders, and planning long-term sustainability alongside software design. The FAIR-USE4OS guidelines will allow funders and researchers to more effectively evaluate and plan open-source software projects. There is good evidence of funders increasingly mandating that all funded research software is open source; however, even under the FAIR guidelines, this could simply mean software released on public repositories with a Zenodo DOI. By creating FAIR-USE software, best practice can be demonstrated from the very beginning of the design process and the software has the greatest chance of success by being impactful.

MOTIVATION: In this article, we consider how to evaluate survival distribution predictions with measures of discrimination. This is non-trivial as discrimination measures are the most commonly used in survival analysis and yet there is no clear method to derive a risk prediction from a distribution prediction. We survey methods proposed in literature and software and consider their respective advantages and disadvantages. RESULTS: Whilst distributions are frequently evaluated by discrimination measures, we find that the method for doing so is rarely described in the literature and often leads to unfair comparisons or 'C-hacking'. We demonstrate by example how simple it can be to manipulate results and use this to argue for better reporting guidelines and transparency in the literature. We recommend that machine learning survival analysis software implements clear transformations between distribution and risk predictions in order to allow more transparent and accessible model evaluation. AVAILABILITY AND IMPLEMENTATION: The code used in the final experiment is available at https://github.com/RaphaelS1/distribution_discrimination.

SUMMARY: As machine learning has become increasingly popular over the last few decades, so too has the number of machine-learning interfaces for implementing these models. Whilst many R libraries exist for machine learning, very few offer extended support for survival analysis. This is problematic considering its importance in fields like medicine, bioinformatics, economics, engineering and more. mlr3proba provides a comprehensive machine-learning interface for survival analysis and connects with mlr3's general model tuning and benchmarking facilities to provide a systematic infrastructure for survival modelling and evaluation. AVAILABILITY AND IMPLEMENTATION: mlr3proba is available under an LGPL-3 licence on CRAN and at https://github.com/mlr-org/mlr3proba, with further documentation at https://mlr3book.mlr-org.com/survival.html.