Simulating a Statistics Head-to-Head Rumble: Round Two

In the previous post, I considered a quasi-Dutch Book scenario inspired by a LinkedIn poster. In the investigation of that scenario, I discovered that the fact that the stakes were fixed and positive, together with the selection bias of the bookie, introduced an apparent bias into the performances of each of the statistical methods: namely the game incentivizes the bookie to take bets where a bettor underestimates the ’true’, long-run value of the bet. The statistical methods themselves weren’t biased in this way, but they simply wouldn’t be chosen by the bookie except in the cases where they happened to do so by chance. This effect skews the previous results as indications of the long-run performance of those methods. In order to correct this tendency and better align with proper Dutch Books, we should introduce the notion of a negative bet for the bookie.

Negative Bets

By negative bet, I mean a procedure whereby, instead of a bettor purchasing a bet at their fair bet price for potential stakes, I mean an equivalent procedure where the bookie gives the bettor the difference between the stakes and their fair bet price under the agreement that the bookie will claw back the entire stakes if the bet fails. (Note that this is identical to how traditional Dutch Books punish subadditive ‘probabilities’ with negative stakes.)

To understand this equivalence, consider the following: Let \(S\) be the stakes. Let \(p\) be the bettor’s estimation of the bias of the coin. Let \(q\) be the ’true’ bias of the coin. Then:

\((S-pS) - (1-q)S\)

\(S-pS-S+qS\)

\(-pS+qS\)

\((q-p)S\)

Where the \((S-pS)\) term is the initial giving or loan by the bookie, and \((1-q)S\) is the long-run average of how often the bookie claws back the stakes.

This last line is the negative of (p-q)S, which is zero at a perfectly calibrated bet. In the previous simulation, we had the case where the bookie tended to take bets where p-q > 0, that is where q underestimated p. In the case of a negative bet, the bookie will be incentivized oppositely, to take bets where q overestimates p.

By allowing the bookie to make positive and negative bets equally randomly (~Bern(0.5)), we further improve the scenario by removing the previous bias introduced by the behavior of the bookie. Note that this alteration makes it one step further from the original scenario considered, but nonetheless hopefully provides a clearer view into the underlying issues at play.

Hypotheses Redux

The hypothesis here is that by introducing negative bets, the selection bias against, for example, the Frequentist Confidence Upper Bound bettor is corrected and that the bookie’s average rate of gain no longer places the Oracle bettor in the middle of performers but instead places it at the lowest position, being the hardest to exploit.

Results and Discussion

Considering the results below, my hypothesis seems at least partially incorrect. While the results obtained indeed seem to correct some of the selection bias in the previous scenario (as depicted in graphs not displayed here), it does not much improve the rankings of FUB in terms of win rate or the Oracle in terms of the bookie gain rate.

I suspect that some selection bias still exists relative to the betting strategies, as the bookie is still able to skip placing bets, but this is the kind of behavior we want in the bookie, so this does not seem entirely eliminable for our purposes – we’re interested in understanding the exploitablility of these strategies, after all.

What we do see, however, is that, again, OB and OB-adjacent methods are highly competitive in terms of head-to-head wins against all other methods, and that OB and OB-adjacent methods are not particularly exploitable when compared to frequentist methods, including the MLE (plug-in).

We also again see the problems that can arise with subjectivist Bayesian methods, which can still perform poorly even with empirical calibration, depending on the extremity of their prior. Most practicing Bayesians today have some rules of thumb against such unwarranted prior confidence, but OB provides a more particular and well-motivated set of desiderata for selection.

Closing Remarks

I’ve found the last few informing experiments to be enlightening and educational. It’s certainly interesting to see which parts of the purports of theory and intuition bear out in practice. As I continue to learn more about various statistical methodologies, I think it’ll revisit such simulations in the future to test long-run expected results, as that criteria is relevant. Recall that subjectivist Bayesian methods are not particularly interested in long-run expected performance as a criterion of evaluation in the first place, so complaining about the poor long-run performance of subjectivist statistical methods is a bit misplaced. But as Bayesian methods make more headway into the sciences, Bayesian practitioners need to recognize that, especially in scientific and industrial contexts, long-run performance is relevant. As such, subjectivist methods may not be appropriate in those cases. This is why contemporary, scientifically-minded Bayesianism has moved in more objectivist directions for the last several decades.

The classic objections to frequentism still stand as strong as ever, however. Frequentist methods offer empirical reliability but they answer the wrong questions in the first place. After all, when Joe gets screened for cancer, he doesn’t want to know how often people like Joe have cancer: he wants to know how likely it is he has cancer. Insofar as frequentist information is used for managing practical decisions, a sleight of hand takes place, yielding implicit Bayesian readings despite being unsupported by frequentist interpretations. Objective Bayesianism provides a coherent framework that makes sense of why this sleight of hand is actually rational and where frequentist information fits within a broader Bayesian rationality. OB allows Bayesians to take up available empirical information that subjectivists leave on the table and allows frequentists to answer the correct questions for decision making without losing their long-run performance bounds.

A relevant question from a practical perspective, though, is that since frequentism is at least approximately correct in most scientific contexts, is there a practical benefit to going through the trouble of producing OB results? That, however, will be a question for another time.

References

• Berger, J., Bernardo, J., & Sun, D. (2024). Objective Bayesian Inference. WORLD SCIENTIFIC.
• Williamson, J. (2010). In defence of objective Bayesianism, Oxford University Press.

Appendix

Head-to-Head Results

Win Counts

Bookie Gain Rates

Average Bookie Gain Rate