Monitoring is an important aspect of safely deploying Large Language Models (LLMs). This paper examines activation probes for detecting ``high-stakes'' interactions---where the text indicates that the interaction might lead to significant harm---as a critical, yet underexplored, target for such monitoring. We evaluate several probe architectures trained on synthetic data, and find them to exhibit robust generalization to diverse, out-of-distribution, real-world data. Probes' performance is comparable to that of prompted or finetuned medium-sized LLM monitors, while offering computational savings of six orders-of-magnitude. These savings are enabled by reusing activations of the model that is being monitored. Our experiments also highlight the potential of building resource-aware hierarchical monitoring systems, where probes serve as an efficient initial filter and flag cases for more expensive downstream analysis. We release our novel synthetic dataset and the codebase at \url{https://github.com/arrrlex/models-under-pressure}.

The applicability and utility of Artificial Intelligence (AI) based solutions has been demonstrated widely in the healthcare domain via the automated analysis of medical information. However, the adoption rate of AI-based healthcare systems is inhibited due to their complicated nature. Also, the hierarchical nature of the medical settings adds a layer of complexity in understanding how a Machine Learning (ML) model functions when subjected to varying workflows. Variation in models' performance, and individual features' contribution, needs to be effectively quantified such that medical practitioners can understand the models, and validate their operation if widespread adoption is to be enabled. In this paper, a hierarchical medical workflow for understanding the operation of ML in a healthcare-based setting is proposed. Its utility is demonstrated in the context of heart disease classification. Explainable Artificial Intelligence (XAI) is incorporated in the form of Feature Importance (FI) scores and correlated with an ML model's performance metrics (Accuracy, F1-score). This provides a multi-stakeholder perspective aligned with the hierarchy as experienced in a real-world medical setting. The paper contributes a methodology for accommodating an enhanced understanding of diverse hierarchical healthcare settings that would benefit from the adoption of AI-based systems.