This paper presents a novel framework for analyzing and designing evaluation metrics in binary classification tasks. Traditional metrics—such as Accuracy, Precision, Recall, F1-score, and Cohen’s —often embed implicit assumptions about the relative costs and benefits of correct and incorrect predictions. However, these assumptions are not always transparent and may not align with domain-specific cost–benefit structures. By systematically evaluating classifiers through an underlying reward matrix, the proposed framework reveals that each metric reduces to a single break-even ratio between the resources invested and the value gained. This connects classical confusion matrix based metrics to an explicit cost–benefit interpretation. We derive this ratio explicitly for several widely used confusion matrix based metrics, thereby making their implicit trade-offs directly comparable under a unified interpretation. The paper demonstrates how metric values can be interpreted and applied to comprehensively assess classifier performance. Additionally, the framework allows researchers to define new metrics tailored to specific problem requirements. Experiments with commonly used metrics illustrate the framework’s broad applicability and highlight the value of explicitly modeling both costs and benefits for more context-sensitive performance evaluation.

Binary classification deals with identifying whether elements belong to one of two possible categories. Various metrics exist to evaluate the performance of such classification systems. It is important to study and contrast these metrics to find the best one for assessing a particular system. Despite extensive research in this field, a particular systematic comparison of these evaluation metrics remains an unaddressed area. The performance of a classifier is usually evaluated through the confusion matrix, a table including the count of accurate and inaccurate predictions for each category. To judge if one classifier is better than another, examining variations in the confusion matrix is necessary. However, no agreed-upon method exists for this analysis. This is crucial because different metrics may interpret and rate two confusion matrices differently. We introduce the Worthiness Benchmark (γ), a new concept useful to characterize the principles by which performance metrics rank classifiers. In particular, the Worthiness Benchmark is useful to assess how a metric evaluates the superiority among two classifiers by analyzing differences in their confusion matrices. Through this new concept, we are able to deal with the main challenge of selecting the best metric to evaluate a classifier. We then perform a γ-analysis on several binary classification metrics to outline the specific benchmarks these metrics follow when comparing different classifiers.