Probabilism and "Accuracy": An Introduction

Jake Metzger

In previous posts I’ve talked about some ways of establishing specifically a Bayesian epistemology as the governing norms of rational inference. There is Cox’s Theorem and its variants of course, but I’ve specifically spent time in Dutch Book Arguments which show that if one fails to reason in accordance with the rules of probability, there are practical scenarios under which one can be exploited.

Another set of arguments that I’ve recently started looking into are so called “accuracy” arguments for probabilism. The potentially problematic nomenclature aside, what these sorts of arguments aim to show is that if one’s credences fail to be probabilistically consistent, then there is a probabilistic version of those beliefs that are better, as measured by usual performance metrics, such as a Brier score, regardless of the truth of the propositions under consideration. This provides some independently motivated justification for preferring probabilistically consistent assignments of credence.

Measuring the Performance of Credences

Consider a proposition X and let C be a (potentially non-probabilistic) credence function on propositions. That is, C is some function from propositions to real numbers. Let us say that C(X) = 0 when we believe that X is false and C(X) = 1 when we believe X is true. Otherwise, C can be general.

With regard to a proposition X, we can consider various credence assignments to X and ~X (the negation of X), as seen below. Each red point is a potential credence pair C(X), C(~X). The blue points represent the two possible eventual belief states regarding X and ~X once the truth is revealed. The orange line between them is the set of probabilistically consistent belief states regarding the propositions X and ~X. The dotted red lines with green x’s are the projections of probabilistically incoherent states into the nearest coherent probabilistic state (usually an orthogonal projection).

Now, let’s consider the Brier score. The Brier score is the mean squared error between a probabilistic prediction and the truth. Note that in this case, the act of minimizing the Brier score is equivalent to minimizing the Euclidean distance between one of our red points (our credences about propositions X and ~X) and one of the truth points (0, 1) or (1, 0).

The contention is that probabilistically consistent predictions (those green x’s on the orange line) minimize the Brier score between one’s credences and the truth regardless of what the truth turns out to be. This is because whenever one chooses a probabilistically incoherent set of credences, there is always a probabilistically coherent projection of those credences that has a lower Brier score.

In the diagram above, it should be immediately obvious why this is. We can represent the Brier score (up to monotonicity) as a series of concentric circles emanating from each of the blue “truth” points. For every non-probabilistic set of credences, such circles necessarily hit their probabilistically coherent projections first (the projections on the line y=-x+1 in [0,1]).

In terms of the “accuracy” argument, this shows that incoherent credences are always “accuracy-dominated” by a set of coherent ones.

Does this Depend on the Brier Score specifically?

Now, one question would be, how much of this depends on the use of the Brier score? Sure, the Brier score is one exceedingly common way to measure the “accuracy” of a predictor, but it isn’t the only way. After all, why should we prefer mean squared distance?

The thing is that this holds not just for the Brier score, but for a variety of Lp-norms. The Brier score is an incarnation of the Lp-norm where p=2. Those of us that work in machine learning are familiar with the geometries of various Lp-norms, which are also demonstrated in the following for p>=1:

Figure 1: By Quartl - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=17428655

Figure 1: By Quartl - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=17428655

We can easily see by the diagonal symmetry of these norms and the convexity for p>=1 that for all Lp-norms with p>1 that all of the probabilistically consistent credences will have lower norm from the truth, regardless of which of the blue points it turns out to be. Equality only occurs here when p=1, which indicates that only in this case do non-probabilistic credences eek parity with probabilistic ones.

One might then ask, “What about Lp norms where p<1?”. Indeed, when p<1, we can start to see cases where the off-diagonal, non-probabilistic credences have lower scores than the probabilistic ones. But we should ask whether the Lp norms make sense when p<1 for our case? I think not.

While I’m not convinced that one should necessarily accept the Brier score as the score to use in this case (hence all the discussion about other Lp norms), I do think that non-convex Lp norms are inappropriate here. Indeed, the concave Lp norms display what I take to be an objectionable preference for setting one or other of the credence pairs identically to zero. On the flip side, the convex Lp norms display some preference for keeping both options open to some extent, nonzero.

One might be inclined to say that preferring convexity in the Lp norm is question-begging against the non-probabilist, since this rules out the p<1 cases. But I retort that it is not the convexity of the Lp norm (p>=1) that is the problem for the non-probabilist, but the symmetry of these norms. However, I take it for granted that symmetry in our metric is an obvious requirement, since otherwise opens up objectionable degrees of arbitrariness that the non-probabilist should not find palatable.

Conclusion

Overall, I think this sort of “accuracy” argument is pretty interesting so far, and I certainly need to do more reading and discussion here. I do think the naming is terrible, but it’s the naming that’s stuck apparently. However the considerations themselves are straightforward and, at least on the surface, fairly compelling. What we’ve seen is that we consider a performance metric such as the Brier score to measure the performance of our (potentially non-probabilistic) credences in matching the truth, then whatever the performance of a non-probabilistic set of credences, it is always dominated by the performance of some probabilistically consistent set of credences. And that this is true over not just the Brier score in particular, but over Lp-norms with p>=1 (p=1 being a weak dominance). [In the future I’d like to evaluate just which other kinds of calibration norms in the literature also support this sort of argument.]

I also don’t think that it’s any surprise that we’re seeing an important relationship between the truth and rational norms surrounding convex sets of beliefs about those truths. As I’ve discussed in other posts, Williamson takes building convex sets of beliefs to be essential in incorporating and bounding rational belief under, for example, calibration constraints. I haven’t explored anything deep here, but I do think it’s more than a coincidence that the (minimal) convex set incorporating total credence in X and total credence in ~X is exactly the probabilistically consistent credence pairs.

Resources

(I wrote the above without reading these resources below yet (aside from the videos), but these are where I believe this sort of argument to stem, and I’ll consume them imminently.)

On-topic:

After reading: Interestingly, Joyce provides reasons for rejecting Lp-norms other than n=2 saying that they violate his notion of symmetry. I need to unpack more about how my casual reference to symmetry here differs from his notion of symmetry. I initially thought they would be the same, but his comments about Lp surprised me a bit, but I suppose I’m only yet considering (X, ~X) rather than (X,Y) and such. He also provides some interesting considerations in favor of convexity, which I generally agree with so far, which I think bolster what I said above for p<1. Need to digest more, though.

Looks particularly interesting given that it touches in areas of particular interest to the Objective Bayesian, namely aligning credence with evidence in the context of Joyce’s accuracy argument.