Advances in artificial intelligence (AI) have enabled unprecedented capabilities, yet innovation teams struggle when envisioning AI concepts. Data science teams think of innovations users do not want, while domain experts think of innovations that cannot be built. A lack of effective ideation seems to be a breakdown point. How might multidisciplinary teams identify buildable and desirable use cases? This paper presents a first hand account of ideating AI concepts to improve critical care medicine. As a team of data scientists, clinicians, and HCI researchers, we conducted a series of design workshops to explore more effective approaches to AI concept ideation and problem formulation. We detail our process, the challenges we encountered, and practices and artifacts that proved effective. We discuss the research implications for improved collaboration and stakeholder engagement, and discuss the role HCI might play in reducing the high failure rate experienced in AI innovation.

Clinical practice guidelines, care pathways, and protocols are designed to support evidence-based practices for clinicians; however, their adoption remains a challenge. We set out to investigate why clinicians deviate from the “Wake Up and Breathe” protocol, an evidence-based guideline for liberating patients from mechanical ventilation in the intensive care unit (ICU). We conducted over 40 hours of direct observations of live clinical workflows, 17 interviews with frontline care providers, and 4 co-design workshops at three different medical intensive care units. Our findings indicate that unlike prior literature suggests, disagreement with the protocol is not a substantial barrier to adoption. Instead, the uncertainty surrounding the application of the protocol for individual patients leads clinicians to deprioritize adoption in favor of tasks where they have high certainty. Reflecting on these insights, we identify opportunities for technical systems to help clinicians in effectively executing the protocol and discuss future directions for HCI research to support the integration of protocols into clinical practice in complex, team-based healthcare settings.

Artificial intelligence (AI) in healthcare has the potential to improve patient outcomes, but clinician acceptance remains a critical barrier. We developed a novel decision support interface that provides interpretable treatment recommendations for sepsis, a life-threatening condition in which decisional uncertainty is common, treatment practices vary widely, and poor outcomes can occur even with optimal decisions. This system formed the basis of a mixed-methods study in which 24 intensive care clinicians made AI-assisted decisions on real patient cases. We found that explanations generally increased confidence in the AI, but concordance with specific recommendations varied beyond the binary acceptance or rejection described in prior work. Although clinicians sometimes ignored or trusted the AI, they also often prioritized aspects of the recommendations to follow, reject, or delay in a process we term “negotiation.” These results reveal novel barriers to adoption of treatment-focused AI tools and suggest ways to better support differing clinician perspectives.