Confidence, Calibration, and the Belief You Can't Escape - A Moorean Challenge
Jake Metzger
- Introduction
- The Moorean Tension
- Betting, Belief, and the Single Case
- Can Calibration Replace Belief?
- Misunderstanding Bayesian Inference
- What Counts as Inference?
- Conclusion: Betting, Belief, and Epistemic Accountability
Introduction
In a recent discussion1, I found myself pressing on a tension in one particular interpretation of confidence intervals - namely, the claim that they can guide betting behavior in the single case, even while one denies any belief that the interval actually contains the parameter of interest.
This view was put forward by a self-described frequentist, though I suspect it’s not representative of all (or even most) frequentist thinking. Still, it’s a revealing case study because it highlights a deep fault line between frequentist and Bayesian paradigms: the relationship between probability, rational belief, and action. In what follows, I’ll argue that treating confidence intervals as betting instruments - without any corresponding belief - leads to a Moorean-style incoherence in rational behavior. And attempts to avoid that incoherence end up importing Bayesian commitments through the back door.
The Moorean Tension
Suppose you say, “I’ll bet $100 at 19:1 odds that the parameter lies in this interval, because that’s what the procedure says to do” - and in the next breath say, “But I don’t believe the parameter is in the interval.”
That’s not a contradiction. But it is epistemically bizarre - like saying, “This gun is probably loaded, so I’m pulling the trigger - but I don’t believe it’s loaded.” These are classic Moorean statements: not logically contradictory, but they violate norms of rational belief and action. Without proposing any particular resolution to Moorean statements, it’s enough to recognize that they are widely understood to be problematic from a deliberative, action-oriented perspective. If you’re taking a bet that presumes something is true, yet simultaneously denying that you believe it’s true, then your behavior becomes epistemically unmoored.
Now, one could respond: “I’m not betting because I believe - it’s just policy. I follow this rule because it works well over time.” But that just shifts the problem. If your bet is merely a product of procedural regularity, not an expression of belief or credence, then the bet has lost its evidential interpretation. You’re not really betting on the parameter being in the interval - you’re just executing a rule.
This undermines the original point, which was to justify confidence intervals as guides to action. Betting based on long-run performance without belief strips the bet of any epistemic content. The Moorean tension arises precisely when someone tries to combine a frequentist error-control rationale with a single-case decision, while disavowing any corresponding doxastic commitment. It’s not frequentism per se that causes the problem - it’s the attempt to stretch it into a decision-theoretic role it doesn’t naturally support.
Betting, Belief, and the Single Case
The heart of the issue is this: confidence intervals are frequentist objects. They are designed to have a long-run coverage property under repeated sampling - not to license probability claims about a particular case.
So how do you justify betting on one?
Here’s the best steelman I can offer: “I used a procedure that covers the true parameter 95% of the time. I now have a specific confidence interval from that procedure. Since I have no special information that would mark this case as unusual, I’ll act as if there’s a 95% chance this interval contains the parameter.”
This line of reasoning is familiar. But it doesn’t rescue the interpretation - it just reframes it in Bayesian terms. The idea that this CI is “typical” relies on a symmetry or exchangeability assumption. And that assumption /is not part of the frequentist apparatus/. It’s an epistemic stance - an implicit prior - that licenses you to treat long-run frequencies as subjective probabilities in the single case.
In fact, the reasoning closely mirrors David Lewis’s Principal Principle, which says: if you know the objective chance of some proposition is x, and you lack inadmissible information, then your degree of belief should also be x. In other words, if you know the objective chance of a proposition and have no disqualifying information, then your subjective belief should match that chance. But the Principal Principle belongs to a broadly Bayesian picture. If you’re betting as though your interval has a 95% chance of being right, then unless you can derive that credence from something other than long-run frequency and ignorance, you’re already reasoning like a Bayesian.
There’s nothing wrong with that! But let’s not pretend it’s purely frequentist. The moment you assign a probability to a specific proposition about a fixed but unknown parameter, you’ve left the frequentist safe zone.
To be fair, some frequentist frameworks - like game-theoretic probability (Shafer and Vovk) or minimax bounds - try to guide action without invoking belief. But they only succeed within tightly constrained models where assumptions about the world are baked into the protocol. They don’t actually escape epistemic commitments; they just hide them in the setup. When applied to real-world inference, these frameworks either smuggle in Bayesian reasoning or leave the decision-maker rudderless. And when those systems do produce actionable forecasts, it’s because the environment has stable patterns - which is just another way of saying the forecaster believes in structure.
Can Calibration Replace Belief?
A likely objection goes something like this:
I’m not expressing belief. I’m calibrating my behavior. I follow a rule that performs well in repeated trials. That’s rational, even if I assign no credence to this specific case.
This is more sophisticated than the earlier objection. It invokes /reliability/ rather than belief - bet as a form of performance tracking, not doxastic commitment.
But it still doesn’t work. Why? Because if your betting policy is based on performance in past trials, you’re implicitly assuming that past and present cases are exchangeable. That is: you believe this case is drawn from the same population as those past ones. And /that/ is a belief. It’s a substantive assumption that links long-run reliability to rational action in the present. Without that link, your behavior is just habit. Without that belief, you’re not calibrated - you’re just automated.
Some might insist this doesn’t require belief - that following a high-performing rule is simply rational behavior, even if we withhold judgment about this case. But even instrumental rationality depends on assumptions - namely, that the rule’s past success is predictive of its performance now. If we bracket that assumption, we lose the bridge between past performance and current action. That bridge is epistemic, whether we acknowledge it or not. If you’re using a rule purely because it performs well, you still need to explain why that past performance licenses action now. And unless you assume this case is similar in relevant ways to past cases, the link between policy and rational action breaks. That assumption - of exchangeability or typicality - is epistemic. Without it, you’re not instrumental; you’re just reflexive.
So while “calibration without belief” sounds attractive, it can’t get off the ground without at least /some/ epistemic commitment to the typicality of the current case. And that’s just Bayesianism under another name.
Misunderstanding Bayesian Inference
During our exchange, my interlocutor accused Bayesianism of being an “opinion-assignment game,” where degrees of belief are arbitrary, unfalsifiable, and unscientific. This is a tired caricature.
Bayesian inference is //not// unconstrained. It is governed by coherence (e.g., avoiding Dutch books), decision theory, proper scoring rules, and empirical calibration. Priors can be subjective, yes - but that doesn’t make them unprincipled. And more importantly, Bayesian models are /predictive/. They make testable claims. The belief isn’t the evidence - it’s what follows /from/ the evidence, via a formal updating regime.
And let’s be honest: single-case frequentist claims aren’t falsifiable either. You can’t empirically verify whether this specific confidence interval covers the parameter. So appeals to Popperian falsifiability are either selective or misapplied. If we’re serious about evaluating inference methods, we should focus on predictive performance and decision quality - on both of which Bayesian methods hold up just fine.
Reject Bayesianism if you must - but reject it for what it actually is, not a strawman.
What Counts as Inference?
At one point, my opponent suggested that being logically consistent isn’t enough to count as a theory of inference. I agree. But he never offered a coherent alternative.
Here’s what seems to be the underlying confusion: frequentist methods are /procedural/. They define error rates and coverage guarantees over hypothetical repetitions. But inference isn’t just about procedure - it’s about what follows from data. It’s about what we should believe, or do, given the evidence. And on that front, frequentism is often silent - unless it quietly imports beliefs about model specification, relevance, or typicality.
Frequentist methods often work well in practice - but partly because users smuggle in Bayesian reasoning. Assumptions about model adequacy, plausibility, and relevance aren’t part of the frequentist formalism. They’re epistemic extras, and they undermine the claim to objectivity.
Conclusion: Betting, Belief, and Epistemic Accountability
If you’re going to treat confidence intervals as betting instruments, then be honest about what that entails. You must either:
- Acknowledge the beliefs that justify betting behavior, or
- Explain how betting can be rational without belief.
So far, no purely frequentist story has made option (2) work. Calibration-based defenses quietly rely on beliefs about typicality. Procedural defenses sever the link to inference. And the idea of “confidence without belief” falls into the Moorean trap.
Meanwhile, if you’re going to critique Bayesianism, do it carefully. Bayesian inference is a formal, predictive, coherent system. It’s not “just making stuff up.” And frequentist methods, as practiced, often lean on Bayesian interpretations anyway.
Inference isn’t just about math: it’s about how we turn data into action and whether we’re willing to stand behind the bets we make. If you’re going to act - especially if you’re going to bet - then you’d better be prepared to say what you believe. Otherwise, your actions are just rituals.
The Moorean tension isn’t just an oddity: it’s a warning. If your behavior demands belief but your theory denies it, something has to give.
Just saying “try it and see” isn’t going to cut it.
Footnotes
1 Obligatory LinkedIn post