• No single scoring rule best predicts Lockean accuracy across all belief thresholds. • The Spherical score is optimal for lower thresholds; the Power3 rule is best for higher thresholds. • The widely used Brier and log scores rarely perform best for predicting categorical accuracy. • Most scoring rules work by rewarding calibration and sharpness, but some track additional components. • Empirical methods can complement a priori arguments in evaluating scoring rules. The debate about which scoring rule best measures the accuracy of our credences has largely been conducted on an a priori basis. We pursue an empirical approach, asking which rule best predicts a practical, decision-relevant criterion: Lockean accuracy, the ability to make correct categorical judgments based on a threshold of belief. Analyzing a large dataset of probability judgments, we compare the most widely used scoring rules (Brier, logarithmic, spherical, absolute error, and power rules) and find that, among them, there is no single best one. Instead, the optimal choice is context dependent: the Spherical score is the best predictor for lower belief thresholds, while the Power 3 rule is best at higher thresholds. In particular, the widely used Brier and log scores are rarely optimal for this task. A mediation analysis reveals that while much of a rule’s success is explained by its ability to reward calibration and sharpness, the Spherical and Brier rules retain significant predictive power independently of these standard virtues.

According to the Lockean thesis (LT), a proposition is believed just in case it is highly probable. While this thesis enjoys strong intuitive support, it is known to conflict with seemingly plausible logical constraints on our beliefs. One way out of this conflict is to make probability 1 a requirement for belief, but most have rejected this option for entailing what they see as an untenable skepticism. Recently, two new solutions to the conflict have been proposed that are alleged to be non-skeptical. We compare these proposals with each other and with the LT, in particular with regard to the question of how much we gain by adopting any one of them instead of the probability 1 requirement, that is of how likely it is that one believes more than the things one is fully certain of.