The Two Paper Rule

A repost from my old blog.

I’ve been thinking a lot these days about the danger of “credentialist cartels”. This is a situation where outsiders aren’t allowed to challenge the ideas of the credentialed insiders in a field, but where the insiders are forced to defend those ideas by the necessity of maintaining the prestige of their credentials. I’m going to write about this more in the coming days, but first I thought I’d re-up a post from my old blog that deals with an important aspect of this question.

The original title of this post was “Vast literatures as mud moats”. It’s about a dilemma faced by anyone who wants to engage with a field from the outside: How much of the literature do I have to read in order to know if the field is even worth engaging with? In the realm of public debate, we often encounter paradigms that, at first glance, seem like they might be quackery and might have something valuable to say — MMT, evolutionary economics, or Austrian economics, to name just a few in the econ world. People are always coming up to pundits like me and yelling that if we just immersed ourselves in their literature for a few years, we’d realize that they’re right about everything. Of course we don’t have years to spare. But if we simply cede the field whenever anyone comes along claiming that their literature gives them deep expertise, we make our body politic vulnerable to cults, scam artists, and quacks.

So in this post from 2017, I suggest a method to very quickly tell if a school of thought is worth paying attention to. It has served me quite well over the years. I call it the Two Paper Rule.

---

I don't know why academic literatures are so often referred to as "vast" (the phrase goes back well over a century), but it seems like no matter what topic you talk about, someone is always popping up to inform you that there is a "vast literature" on the topic already. This often serves to shut down debate, because it amounts to a demand that before you talk about something, you need to go read voluminous amounts of what others have already written about it. Since vast literatures take many, many hours to read, this represents a significant demand of time and effort. If the vast literature comprises 40 papers, each of which takes an hour to read, that's one week of full-time work equivalent that people are demanding as a cost of entry just to participate in a debate! So the question is: Is it worth it?

Often, reading the literature seems like an eminently reasonable demand. Suppose I were to think about minimum wage for the first time, knowing nothing about all the research economists have done on the topic. I might very confidently say some very silly things. I would be unaware of the relevant empirical evidence. There would probably be theoretical considerations I hadn't yet considered. Reading the vast literature would make me aware of many of these. In fact, I think the minimum wage debate does suffer from a lack of knowledge of the literature.

But the demand to "go read the vast literature" could also be eminently unreasonable. Just because a lot of papers have been written about something doesn't mean that anyone knows anything about it. There's no law of the Universe stating that a PDF with an abstract and a standard LaTeX font contains any new knowledge, any unique knowledge, or, in fact, any knowledge whatsoever. So the same is true of 100 such PDFs, or 1000. 

There are actual examples of vast literatures that contain zero knowledge: Astrology, for instance. People have written so much about astrology that I bet you could spend decades reading what they've written and not even come close to the end. But at the end of the day, the only thing you'd know more about is the mindset of people who write about astrology. Because astrology is total and utter bunk.

But astrology generally isn't worth talking or thinking about, either. The real question is whether there are interesting, worthwhile topics where reading the vast literature would be counterproductive - in other words, where the vast literature actually contains more misinformation than information.

There are areas where I suspect this might be the case. Let's take the obvious example that everyone loves: Business cycles. Business cycles are obviously something worth talking about and worth knowing about. But suppose you were to go read all the stuff that economists had written about business cycles in the 1960s. A huge amount of it would be subject to the Lucas Critique. Everyone agrees now that a lot of that old stuff, probably most of it, has major flaws. It probably contains some real knowledge, but it contains so much wrong stuff that if you were to read it thinking "This vast literature contains a lot of useful information that I should know," you'd probably come out less informed than you went in. 

Of course, many would say the exact same about the business cycle theory literature that emerged in response to the Lucas Critique and continues to this day. But if so, that just makes my point stronger. The point is, a bunch of smart people can get very big things wrong for a very long period of time, and that period of time may include the present.

I have personally encountered situations where I felt that reading the vast literature didn't improve my knowledge of the real thing that the literature was about. For example, I read a lot of the macro models that came out in the years following the 2008 financial crisis. Obviously, the financial sector is very important for the macroeconomy (as more people should have realized before 2008, but which almost everyone realizes now). But the ways that macro papers have modeled financial frictions are pretty unsatisfying. They are hard to estimate, the mechanisms are often implausible, and I bet that most or all will have glaring inconsistencies with micro data. I could be wrong about this, of course, but I felt like reading this vast literature was setting me on the wrong track. I'm not the only one who feels this way, either.

The next question is: Can a misinformative vast literature be used intentionally as a tactic to win political debates? It seems to me that in principle it could. Suppose you and your friends wanted to push a weak argument for political purposes. You could all write a bunch of papers about it, with abstracts and numbered sections and bibliographies and everything. You could cite each other's papers. If you wanted to, you could even create a journal, and have a peer review system where you give positive reviews to each other's B.S. papers. Voila - a peer-reviewed literature chock full of misinformation.

In practice, I doubt anyone ever does this intentionally. It takes too much coordination and long-term planning. But I wonder if this sometimes happens by accident, due to the evolutionary pressures of the political, intellectual, and academic worlds. The academic world gives people an incentive to write lots of papers. The political world gives people an incentive to use papers to push their arguments. So if there's a fundamentally bad argument that many people embrace for political reasons, there's an incentive for academics (or would-be academics) to contribute to a vast literature that is used to push that bad argument.

And in the world of intellectual debate, this vast literature can function as a mud moat. That is a term I just made up, sticking with the metaphor of political arguments as medieval castles requiring a defense. A mud moat is just a big pit of mud surrounding your castle, causing an attacking army to get trapped in the mud while you pepper them with arrows.

If you and your buddies have a political argument, a vast literature can help you defend your argument even if it's filled with vague theory, sloppy bad empirics, arguments from authority, and other crap. If someone smart comes along and tries to tell you you're wrong about something, just demand huffily that she go read the vast literature before she presumes to get involved in the debate. Chances are she'll just quit the argument and go home, unwilling to pay the effort cost of wading through dozens of crappy papers. And if she persists in the argument without reading the vast literature, you can just denounce her as uninformed and willfully ignorant. Even if she does decide to pay the cost and read the crappy vast literature, you have extra time to make your arguments while she's so occupied. And you can also bog her down in arguments over the minute details of this or that crappy paper while you continue to advance your overall thesis to the masses.

So when I want to talk and think and argue about an issue, and someone says "How about you go read the vast literature on this topic first?", I'm presented with a dilemma. On one hand, reading the vast literature might in fact improve my knowledge. On the other hand, it might be a waste of time. And even worse, it might be a trap - I might be charging headlong into a rhetoritician's mud moat. But choosing not to read the vast literature keeps me vulnerable to charges of ignorance. And I'll never really be able to dismiss those charges.

My solution to this problem is what I call the Two Paper Rule. If you want me to read the vast literature, cite me two papers that are exemplars and paragons of that literature. Foundational papers, key recent innovations - whatever you like (but no review papers or summaries). Just two. I will read them. 

If these two papers are full of mistakes and bad reasoning, I will feel free to skip the rest of the vast literature. Because if that's the best you can do, I've seen enough.

If these two papers contain little or no original work, and merely link to other papers, I will also feel free to skip the rest of the vast literature. Because you could have just referred me to the papers cited, instead of making me go through an extra layer, I will assume your vast literature is likely to be a mud moat.

And if you can't cite two papers that serve as paragons or exemplars of the vast literature, it means that the knowledge contained in that vast literature must be very diffuse and sparse. Which means it has a high likelihood of being a mud moat.

The Two Paper Rule is therefore an effective counter to the mud moat defense. Castle defenders will of course protest "But he only read two papers, and now he thinks he knows everything!". But that protest will ring hollow, because if you can show bystanders why the two exemplar papers are bad, few bystanders will expect you to read further.

If it proves to be as effective as I think, the Two Paper Rule, if widely implemented, could make for much more productive public debate. The mud moat defense would be almost entirely neutralized, dramatically reducing the incentive for the production of vast low-quality literatures for political ends. It could allow educated outsiders and smart laypeople access to debates previously dominated by vested insiders. In other words, it could shine the light of reason on a lot of dark, unexplored corners of the intellectual universe.


Update

Some people seem to misunderstand the purpose of the Two Paper Rule. The Two Paper Rule is not about summarizing the literature's findings - for that, you'd want a survey paper or meta-analysis. It's about evaluating the quality of the literature's methodology.

Sometimes a lit review will reveal pervasive methodological weakness - for example, if a literature is mostly just a bunch of correlation studies with no attention to causal effects. But often, it won't. For example, if the literature has a lot of mathematical theory in it, a lit review will generally contain at most one stripped-down partial model. But that doesn't give you nearly as much info about the quality of the fully specified models as you'll get from looking at one or two flagship theory papers. Or suppose a literature consists mostly of literary theorizing; the quality of the best papers will depend on the clarity of the writing, which a lit review is unlikely to be able to reproduce. Sometimes, lit reviews simply report results, without paying attention to what turn out to be glaring methodological flaws.

In other words, if you suspect that a literature functions mainly as a mud moat, what you need to assess quickly is not what the literature claims to find, but whether those claims are generally credible. And that is why you need to see the best examples the literature has to offer. Hence the Two Paper Rule.

Meanwhile, Paul Krugman endorses the Two Paper Rule. Tyler Cowen is more skeptical of its universality.  

---