Unsolicited Opinion on "Frequentist Pursuit"

Jake Metzger

I was watching a YouTube series on Bayesian causal inference, when I came across a presentation by Larry Wasserman [1] raising what he sees as problems with causal inference using Bayesianism.

What it boiled down to, however, is a rather mundane observation that I’ve discussed in previous posts: Bayesian statistics, as it is commonly practiced, doesn’t guarantee frequentist coverage. And, since he thinks that coverage is an important property for scientific (causal) inference, he puts forward that we should find a problem with a Bayesian approach to causal inference.

We’ve discussed this before, and I’ll discuss it again, that Bayesian statistics, as is commonly practiced, is an outgrowth of a subjectivist Bayesian approach to epistemology, with a certain fundamentalism about Bayesian conditionalization. As such, it doesn’t have a place to care about preserving frequency properties. So Wasserman’s observation, while correct, has essentially very little to do with causal inference itself and isn’t a particularly deep observation, in my opinion.

In the discussion section of the talk, he quips about a phenomenon he’s called “frequentist pursuit”. I’ve managed to find a few other discussions related to this concept, but it’s essentially the idea that: If I’m going to gerrymander my Bayesian inference to make it have frequentist properties, why not just do frequentist inference in the first place? Mayo quips that this might be a case of envy by the Bayesian rather than pursuit. [3] Wasserman, in a kind of diplomacy, goes on to say that one should pick the right tool for the job: if one wants frequentist properties, use frequentist inference; if one wants Bayesian properties, use Bayesian inference. [1, 2] However, these perspectives from Wasserman and Mayo just strike me as silly. Color me naive to think so against these heavyweights, but I’ll explain why I think this way.

Statistical inference, on my view, is a way of acquiring knowledge about the world, situated very much complimentarily with things like logical deduction. And responsible knowledge acquisition, organization, and dissemination is governed, at root, by epistemological norms. So, statistical inference, in my view, should be understood in relation to an underlying epistemology for which it is brought into service. In my opinion, frequentism and the results of frequentist inference have a much harder time being situated into a general epistemology, especially a quantitative one. Perhaps one can sit tight with having an ad hoc epistemology specifically for scientific inference that’s different for mundane, everyday inference, but I find this distasteful and problematic. Indeed the quasi-falsificationism that comes out of things like NHST is just not operational for people going about their lives conducting what appears to be responsible and rational inference without null hypotheses and the like. The questions we often ask are not well suited for frequentist norms and, if they can be expressed in frequentist terms at all, yield answers that are not very informative for the original question. If John wants to know if he has cancer based on a screening test, the frequentist has to situate John as a member of some group over which the test has a certain performance. The frequentist answer to whether John has cancer is, “People like John have cancer x% of the time.” But the question was not about “people like John”, it’s about John. Further, the coverage intervals that Bayesians are supposedly envious of don’t answer whether a parameter’s true value is likely within the interval – they attach the performance guarantee to the interval-generating mechanism (under the also-assumed null hypothesis) rather than to the interval generated. In order to infer that “X has a 95% chance of being in [-1, 1]”, a form of question we typically actually care about, we must Bayesianize our interpretation of the interval. The frequentist interpretation of the interval, saying only that the interval-generator has a 95% success rate and it happened to generate [-1, 1] this time, leaves us high and dry when it comes to what we’re supposed to do with that information: I don’t care how accurate the shotgun is in general; I want to know how likely it is that the target was shot when I pulled the trigger.

It’s also worth noting that at least for the past 50 years there have been intramural discussions by Bayesians with regard to the recognition and incorporation of frequency evidence, perhaps the most well-known form of such a principle being David Lewis’ Principal Principle. This principle states, informally, that if one believes that the objective chances of X are such-and-so, then absent other qualifying information, one should have credence such-and-so in X. That is, if I’m told that a coin has an objective bias of 0.7 for Heads, then I should believe that p(Heads) is 0.7. This principle is not a statistical principle but an epistemological one: it governs what is allowable by rational norms. Such a principle is one example of what I call a “bridge principle” which allows for one kind of probability, in this case a frequentist probability, to be transformed into another, such as a degree of belief. Bayesians, at least of the epistemological sort (as opposed to the merely statistical sort), have argued for various such norms using game-theoretic arguments, axiomatic approaches, and others. Worth noting, however, is that, at least as far as I am aware, Bayesians can make use of frequentist information but frequentists continue to be unable to use Bayesian information in general.

Returning back to the concept of frequentist pursuit or frequentist envy, there are a few questions. First, is it problematic for a Bayesian to make use of frequentist statistical methods? The answer to this is a pretty clear “no” for me. Not only are many frequentist methods implicitly Bayesian under certain interpretations but I find it more and more absurd for Bayesians to leave frequency information unaccounted for in their inference. I’m not tied to the particular use of “Bayes”-branded statistical machinery, as long as we’re making adequate use of all of the available information and the results satisfy relevant epistemological norms, like having coherent degrees of belief. My position is that frequentist inference is incomplete in focusing entirely on issues of calibration and error control. Responsible and rational, objective belief is more than this, though frequency information and frequentist behaviors may be primary for a good portion of inference in the sciences.

Second, if Bayesian statisticians start incorporating frequentist information into their inference procedures, is this an indication of pursuit or envy? I don’t think so. My tentative position is that subjectivist Bayesians have been running the hen house for too long, and it seems clear that, particularly for the sciences, frequency information is important. It’s inconsistent to me that the popular Bayesian methods today leave frequency information on the table as though it’s irrelevant. That is, they assume that all information relevant for inference enters into the belief function via conditionalization, particularly in setting the likelihood function according to the empirical data. This is an incomplete application of the Principal Principle, since frequentist inference itself gives us more than just the value of the likelihood for the data. But, since frequentist information isn’t something that can be plugged into Bayes’ Rule, this is overlooked or excluded. It’s my position that it’s most responsible to use all available information to conduct inference, even if it means stepping outside of Bayes’ Rule to appropriately account for it. Is this envy? No, this is responsibility according to a governing epistemology that is quite robustly Bayesian. Meanwhile, frequentist inference (at least NHST) doesn’t respect knowledge of even logical dichotomies, as demonstrated by traditional sure-loss arguments.

Third, why not just be a frequentist? Indeed, especially in online discourse, if one allows that it’s not irrational to use frequentist methods for inference, the question inevitably becomes why one shouldn’t just be a frequentist. After all, frequentist methods were devised to support the kinds of inferences about properties frequentists have tended to care about, just as Bayesian statistical inference was developed to support (subjectivist) Bayesian inferences. The answer for me is that objective Bayesianism is a superior epistemological view to both of these approaches to inference and provides principled regard for the long-run properties associated with frequentism along with the probabilistic coherence of more traditional Bayesian views. Granted, conducting objective Bayesian inference can be more complicated than either frequentist or (subj) Bayesian inference, but deviations from the OB ideal can be justified by making such deviations, and the consequences of such deviations, explicit.

In summary, it’s reductionist to say that if one wants some semblance of “frequentist” properties that one should just use frequentist methods or, worse, become a frequentist. Frequentism, as a philosophy, has a host of its own problems, and frequentist inference is fraught with issues integrating with a broader epistemological view. However, frequentists don’t own knowledge of frequency properties nor can they bully Bayesians from following the sound, game-theoretic justification that calibration (e.g. Lewis’ Principal Principle) is important, just as we already follow such justifications in favor of the Probabilism that underlies Bayesianism in the first place. Probabilism and calibration, as epistemological norms, are not in conflict for the Bayesian, and we should not be goaded by the frequentist into thinking so. We can be grateful that some of frequentist inference can be operationalized by the Bayesian into a more complete and flexible inferential methodology and then go about our business. Indeed, Mayo’s own “error-statistical” methodologies with severity testing could even be brought under the Bayesian umbrella, a prospect I’m interested to investigate in the future. Indeed if there is any envy, it should be that Bayesians have more inferential resources at their disposal to meet more than just long-run performance goals and can do more for rationality than control error rates.

References

Also see related posts quoting Williamson (2010, 2013)