In today’s post, we’re going to talk a bit about about Dutch Books and do a bit of analysis of a post from a self-described “socratic pedagogue to the statistically minded” on LinkedIn, because of course that’s going to be an accurate moniker. More often than not, I find that these statisticians-with-opinions want to express their philosophical opinions rather than opine on statistics that they’re actually trained with, leading to saying silly things. This is one of those cases, but it’s hopefully instructive.[$]

The Trope

A common trope I see from pop-frequentists is that Bayesian inference doesn’t have any (theoretical or practical) guarantees, where frequentist inference does.

As is often the case with bald statement like that 1) the statement is almost certainly false a priori, and 2) that insofar as it could be true, it relies on unstated presumptions or norms. In this case, it’s very much both.

Frequentist Inferential Guarantees - Error Control

Let’s recall that frequentist inference, insofar as we can address it univocally, is typically concerned with “empirical” probabilities, particularly in relation to physical data generating processes or mechanisms. In this sort of inference, it is often hypothesized that there is some true data generating process out there which is described by a statistical model featuring one or more stochastic variables. These variables are hypothesized to have true parameter values describing the stochastic variables. The goal of frequentist inference, then, is often to yield unbiased estimates of these true parameter values.

A short, March 2024 article from Jon Williamson gives a cute puzzle for Bayesian conditionalization. Since it fits in with my recent reading and posts, I think is worth a brief discussion.

Due to the brevity of the article, I suggest that people read that first, though I’ll paraphrase as I understand it. Any errors will, of course, be mine.

The Red Faces Puzzle

Consider the following propositions:

• X = “There is a fair, six sided die to be rolled.”
• E = “Each face of the die is colored either red, blue, or green.”
• A = “The outcome of rolling the die will be >= 3.”
• R = “When the die is rolled, the uppermost face is red.”

Let us initially accept the hypothesis that conditional beliefs are identical to conditional probabilities (CBCP):

In response to one of my previous blog posts, one of my colleagues asked me for a practical example of how to put some Objective Bayesian (OB) ideas into practice, preferably using existing software. I put together the following walk-through to sketch out one seemingly plausible workflow that employs OB ideas on a simple dataset. Afterward, I offer some thoughts.

import statsmodels.api as sm
import pymc as pm
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.model_selection import train_test_split
import numpy as np


# First, let's generate some example data

rng = np.random.RandomState(314)
n_points = 50
x = 12 * rng.rand(n_points)
# True line: y=3x-17
y = 3 * x - 17 + 9 * rng.randn(n_points)
x_data, y_data = x, y
plt.scatter(x, y);

Our first step here is to generate (frequentist) coverage intervals (i.e. confidence intervals) for the predictions here. For generality in the ML world, let’s do that using conformal prediction. I won’t go into any background on conformal prediction here, but it’s a general way of producing coverage intervals that has recently become popular in the ML world (mostly due to so-called split-conformal prediction). Any method of producing (narrow) coverage intervals should work here.

I came across the following short comment from Larry Wasserman:

Larry Wasserman. “Frequentist Bayes is objective (comment on articles by Berger and by Goldstein).” Bayesian Anal. 1 (3) 451 - 456, September 2006. https://doi.org/10.1214/06-BA116H

In this, Wasserman agrees that objective Bayesianism is, well, objective. (At least insofar as “objective” is a useful description at all in statistics. [1])

I interpret “Objective Bayesian inference” to mean “Bayesian methods that have good frequency properties.” I refer to these methods as Frequentist-Bayes…

Information theory provides a constructive criterion for setting up probability distributions on the basis of partial knowledge, and leads to a type of statistical inference which is called the maximum-entropy estimate. It is the least biased estimate possible on the given information; i.e., it is maximally noncommittal with regard to missing information.

Jaynes, E. T. (1957). Information Theory and Statistical Mechanics. Physical Review, 106(4), 620–630. https://doi.org/10.1103/physrev.106.620

Objective Bayesians have been looking toward Information Theory as a foundation for statistics for over 67 years, starting less than a decade after Shannon’s initial publication of the theory in 1948, yet most practitioners are still taught that Bayesianism must be rooted in subjectivity. While Jaynes’ explication of maximum-entropy inference ignores Calibration (in the sense discussed by Williamson 2010), it is clearly not subjective in the sense of de Finetti or Savage; there is no role for personal judgement in the inference - it’s as fixed as any physical parameter might hope to be, except this parameter is informational.

I was recently involved in a small internet spat on LinkedIn with someone who confidently proclaimed the failure of Bayesian statistics and machine learning because, according to them, Bayesianism is inherently subjective. Since I, like many, suffer from Someone-is-wrong-on-the-internet syndrome, I pointed out that this simply isn’t true, that there are quite objectivist strains of Bayesianism at the ready, and pointed him to relevant scholars. As usually happens in these sorts of spats, rather than looking at the provided evidence, he replied back noting the existence of well-(re)known subjectivist Bayesian Leonard Savage as the authority on the subject of Bayesianism as a whole. (They also had a weird hatred of model averaging and tried to blame that on Bayesianism too, which ended up being a clue to disengage.)

The point of statistical inference is not to produce the right answers with high frequency, but rather to always produce the inferences best supported by the data at hand when combined with existing background knowledge and assumptions. Science is largely not a process of falsifying claims definitively, but rather assigning them probabilities and updating those probabilities in light of observation. This process is endless.

Clayton, A. (2022). Bernoulli’s Fallacy: Statistical illogic and the crisis of modern science. Columbia University Press.

I’m Jake Metzger, and I’m currently an AI researcher with a wide set of interests including: artificial intelligence and machine learning, mathematics, epistemology, AI ethics, metaethics, and the like. This blog will be my opportunity to be wrong on the internet about whatever topic strikes my fancy. Hopefully some of you will find my posts interesting too. I also have a Medium blog, which I utilize sparingly. This personal blog will generally have more rough ideas and sketches. Worked out ideas may be graduated to Medium. Reminder: Posts and links here are not reflective of any positions or opinions of any of my employers or affiliations. Further, like most people, my views on things can change over time and as information becomes available, so generally more recent posts will be more representative of my personal views, but not necessarily so.