1. Introduction
2. The Moorean Tension
3. Betting, Belief, and the Single Case
4. Can Calibration Replace Belief?
5. Misunderstanding Bayesian Inference
6. What Counts as Inference?
7. Conclusion: Betting, Belief, and Epistemic Accountability

Introduction

In a recent discussion¹, I found myself pressing on a tension in one particular interpretation of confidence intervals - namely, the claim that they can guide betting behavior in the single case, even while one denies any belief that the interval actually contains the parameter of interest.

A persistent misconception I encounter - especially among frequentists who conflate performance guarantees with epistemic justification - is the claim that statistical models are “convenient yet perhaps imperfect representations of objective reality."¹ This framing obscures a deeper conceptual split in how different schools of inference understand what modeling is for. Here’s the simple point I want to press: the notion that models are “just representations of the world” bakes in assumptions that Bayesians do not share - and for good reason.

Online frequentists, at least from what I find, often charge that (subjectivist) Bayesians do little more than inject bias into their inferences through the use of a subjective prior; frequentism, however, very often claims asymptotic unbiasedness as a virtue for many of its estimators. This is a point in favor of frequentism, or so the proponents say.

There are two ways to read this as a complaint about Bayesianism: from the internal perspective of a frequentist or from an external, potentially non-frequentist or even Bayesian perspective. This is because the very definition of bias depends on what one considers to be the truth of the matter, the norm governing the correctness of the inference.

In previous posts I’ve talked about some ways of establishing specifically a Bayesian epistemology as the governing norms of rational inference. There is Cox’s Theorem and its variants of course, but I’ve specifically spent time in Dutch Book Arguments which show that if one fails to reason in accordance with the rules of probability, there are practical scenarios under which one can be exploited.

Another set of arguments that I’ve recently started looking into are so called “accuracy” arguments for probabilism. The potentially problematic nomenclature aside, what these sorts of arguments aim to show is that if one’s credences fail to be probabilistically consistent, then there is a probabilistic version of those beliefs that are better, as measured by usual performance metrics, such as a Brier score, regardless of the truth of the propositions under consideration. This provides some independently motivated justification for preferring probabilistically consistent assignments of credence.

For the Objective Bayesian, the Calibration norm means ensuring that inference is consistent with known frequency information. I’ve previously discussed examples from Jon Williamson which do this for point estimation. However, one of the major selling points of Bayesian inference is the presence of a genuine posterior distribution. Here I want to demonstrate a fairly simple method for enforcing a calibration constraint on a posterior distribution using some modern ML packages. I’m sure this has been done elsewhere and better, but I hope it can be illustrative. After a quick walkthrough, I want to speculate on the relationship between calibrated posterior distributions and point estimation with an eye toward Williamson’s approach.

In some sense this post is a follow up from my previous comments on so-called “frequentist pursuit”. Recall that in several previous posts I’ve made it clear that the Objective Bayesian (OB) often makes use of frequentist information in their inferences. In doing so, the OB is charged by the snarky frequentist, “If you’re doing frequentist inference anyway, you should just be a frequentist!”

Of course, the OB isn’t doing frequentist inference even if the OB is interested in certain frequentist properties such as coverage. That is, the OB isn’t performing NHST rejection tests nor are they doing Fisherian maximum likelihood inference. The OB, instead, takes the position that their inference should be maximally informed by all available information when performing their inference.

I was watching a YouTube series on Bayesian causal inference, when I came across a presentation by Larry Wasserman [1] raising what he sees as problems with causal inference using Bayesianism.

What it boiled down to, however, is a rather mundane observation that I’ve discussed in previous posts: Bayesian statistics, as it is commonly practiced, doesn’t guarantee frequentist coverage. And, since he thinks that coverage is an important property for scientific (causal) inference, he puts forward that we should find a problem with a Bayesian approach to causal inference.

In the previous post, I considered a quasi-Dutch Book scenario inspired by a LinkedIn poster. In the investigation of that scenario, I discovered that the fact that the stakes were fixed and positive, together with the selection bias of the bookie, introduced an apparent bias into the performances of each of the statistical methods: namely the game incentivizes the bookie to take bets where a bettor underestimates the ’true’, long-run value of the bet. The statistical methods themselves weren’t biased in this way, but they simply wouldn’t be chosen by the bookie except in the cases where they happened to do so by chance. This effect skews the previous results as indications of the long-run performance of those methods. In order to correct this tendency and better align with proper Dutch Books, we should introduce the notion of a negative bet for the bookie.

In the previous post, I responded to a betting scenario from a LinkedIn-er, and one of my key points was that the long run losses experienced by bettors in the bettor’s market was disproportionately funded by uncalibrated bettors. While I think my reasoning there is pretty good, I wanted to make that point more tangible. So, I took to producing some simulations of this market and how different bettors get exploited. This isn’t going to be particularly in depth, but it should be interesting.

In a previous post I outlined one way of incorporating objective Bayesian constraints into a simple regression problem consisting of a few steps:

• Fit a standard regressor of your choosing.
• Use conformal estimation to obtain coverage bounds.
• Continue with your Bayesian analysis as usual, including constraints for the coverage bounds.

This is fine, but one obvious question is, “I don’t want to have to fit one model just to throw it away and fit the one I actually want.”