OBJECTIVES: To assess the potential to adapt an existing technology regulatory model, namely the Clinical Laboratory Improvement Amendments (CLIA), for clinical artificial intelligence (AI). MATERIALS AND METHODS: We identify overlap in the quality management requirements for laboratory testing and clinical AI. RESULTS: We propose modifications to the CLIA model that could make it suitable for oversight of clinical AI. DISCUSSION: In national discussions of clinical AI, there has been surprisingly little consideration of this longstanding model for local technology oversight. While CLIA was specifically designed for laboratory testing, most of its principles are applicable to other technologies in patient care. CONCLUSION: A CLIA-like approach to regulating clinical AI would be complementary to the more centralized schemes currently under consideration, and it would ensure institutional and professional accountability for the longitudinal quality management of clinical AI.

The primary practice of healthcare artificial intelligence (AI) starts with model development, often using state-of-the-art AI, retrospectively evaluated using metrics lifted from the AI literature like AUROC and DICE score. However, good performance on these metrics may not translate to improved clinical outcomes. Instead, we argue for a better development pipeline constructed by working backward from the end goal of positively impacting clinically relevant outcomes using AI, leading to considerations of causality in model development and validation, and subsequently a better development pipeline. Healthcare AI should be "actionable," and the change in actions induced by AI should improve outcomes. Quantifying the effect of changes in actions on outcomes is causal inference. The development, evaluation, and validation of healthcare AI should therefore account for the causal effect of intervening with the AI on clinically relevant outcomes. Using a causal lens, we make recommendations for key stakeholders at various stages of the healthcare AI pipeline. Our recommendations aim to increase the positive impact of AI on clinical outcomes.

OBJECTIVES: Surface the urgent dilemma that healthcare delivery organizations (HDOs) face navigating the US Food and Drug Administration (FDA) final guidance on the use of clinical decision support (CDS) software. MATERIALS AND METHODS: We use sepsis as a case study to highlight the patient safety and regulatory compliance tradeoffs that 6129 hospitals in the United States must navigate. RESULTS: Sepsis CDS remains in broad, routine use. There is no commercially available sepsis CDS system that is FDA cleared as a medical device. There is no public disclosure of an HDO turning off sepsis CDS due to regulatory compliance concerns. And there is no public disclosure of FDA enforcement action against an HDO for using sepsis CDS that is not cleared as a medical device. DISCUSSION AND CONCLUSION: We present multiple policy interventions that would relieve the current tension to enable HDOs to utilize artificial intelligence to improve patient care while also addressing FDA concerns about product safety, efficacy, and equity.

INTRODUCTION: The pitfalls of label leakage, contamination of model input features with outcome information, are well established. Unfortunately, avoiding label leakage in clinical prediction models requires more nuance than the common advice of applying "no time machine rule." FRAMEWORK: We provide a framework for contemplating whether and when model features pose leakage concerns by considering the cadence, perspective, and applicability of predictions. To ground these concepts, we use real-world clinical models to highlight examples of appropriate and inappropriate label leakage in practice. RECOMMENDATIONS: Finally, we provide recommendations to support clinical and technical stakeholders as they evaluate the leakage tradeoffs associated with model design, development, and implementation decisions. By providing common language and dimensions to consider when designing models, we hope the clinical prediction community will be better prepared to develop statistically valid and clinically useful machine learning models.

In a recent review published by Schwartz et al,1 an interdisciplinary team examines clinician involvement in machine learning research. The review includes 80 studies describing predictive clinical decision support systems (CDSSs) targeting clinicians for prognostic or treatment decision making in the hospital using electronic health record data. The objective of the review is to describe clinician involvement in these 80 studies and to map involvement across Stead’s 5 stages of system design.2 Unfortunately, the review makes 2 assumptions about interdisciplinary collaboration that undermine the analysis and interpretation of results. First, the review relies on a novel, highly constrained definition of clinician involvement. The constraints neglect substantial documentation of interdisciplinary collaboration, resulting in dramatic underestimates of collaboration. Second, the review misinterprets missing data. Studies that do not meet the constrained definition of clinician involvement are assumed to have been conducted without any clinician involvement. We highlight the weaknesses of...