We present a novel approach to calibrating linguistic expressions of certainty, e.g., "Maybe" and "Likely". Unlike prior work that assigns a single score to each certainty phrase, we model uncertainty as distributions over the simplex to capture their semantics more accurately. To accommodate this new representation of certainty, we generalize existing measures of miscalibration and introduce a novel post-hoc calibration method. Leveraging these tools, we analyze the calibration of both humans (e.g., radiologists) and computational models (e.g., language models) and provide interpretable suggestions to improve their calibration.

OBJECTIVE: Leverage electronic health record (EHR) audit logs to develop a machine learning (ML) model that predicts which notes a clinician wants to review when seeing oncology patients. MATERIALS AND METHODS: We trained logistic regression models using note metadata and a Term Frequency Inverse Document Frequency (TF-IDF) text representation. We evaluated performance with precision, recall, F1, AUC, and a clinical qualitative assessment. RESULTS: The metadata only model achieved an AUC 0.930 and the metadata and TF-IDF model an AUC 0.937. Qualitative assessment revealed a need for better text representation and to further customize predictions for the user. DISCUSSION: Our model effectively surfaces the top 10 notes a clinician wants to review when seeing an oncology patient. Further studies can characterize different types of clinician users and better tailor the task for different care settings. CONCLUSION: EHR audit logs can provide important relevance data for training ML models that assist with note-writing in the oncology setting.

Major depressive disorder is a debilitating disease affecting 264 million people worldwide. While many antidepressant medications are available, few clinical guidelines support choosing among them. Decision support tools (DSTs) embodying machine learning models may help improve the treatment selection process, but often fail in clinical practice due to poor system integration.

BACKGROUND: Open notes invite patients and families to read ambulatory visit notes through the patient portal. Little is known about the extent to which they identify and speak up about perceived errors. Understanding the barriers to speaking up can inform quality improvements. OBJECTIVE: To describe patient and family attitudes, experiences, and barriers related to speaking up about perceived serious note errors. METHODS: Mixed method analysis of a 2016 electronic survey of patients and families at 2 northeast US academic medical centers. Participants had active patient portal accounts and at least 1 note available in the preceding 12 months. RESULTS: 6913 adult patients (response rate 28%) and 3672 pediatric families (response rate 17%) completed the survey. In total, 8724/9392 (93%) agreed that reporting mistakes improves patient safety. Among 8648 participants who read a note, 1434 (17%) perceived ≥1 mistake. 627/1434 (44%) reported the mistake was serious and 342/627 (56%) contacted their provider. Participants who self-identified as Black or African American, Asian, "other," or "multiple" race(s) (OR 0.50; 95% CI (0.26,0.97)) or those who reported poorer health (OR 0.58; 95% CI (0.37,0.90)) were each less likely to speak up than white or healthier respondents, respectively. The most common barriers to speaking up were not knowing how to report a mistake (61%) and avoiding perception as a "troublemaker" (34%). Qualitative analysis of 476 free-text suggestions revealed practical recommendations and proposed innovations for partnering with patients and families. CONCLUSIONS: About half of patients and families who perceived a serious mistake in their notes reported it. Identified barriers demonstrate modifiable issues such as establishing clear mechanisms for reporting and more challenging issues such as creating a supportive culture. Respondents offered new ideas for engaging patients and families in improving note accuracy.