Economists like to think of peer review as a gatekeeper of quality. It’s a slow, sometimes painful, but ultimately necessary mechanism that separates publishable ideas from the rest. But what if the gate doesn’t really lead anywhere different? In our paper “Unpacking p-Hacking and Publication Bias” (link), we had the rare opportunity to follow manuscripts through every stage of the process at one top field journal: the Journal of Human Resources (JHR).
In one of the (very) secondary (and perhaps most interesting set of) results of the paper, we look at a simple question: where do papers go before and after JHR? The “after” is more interesting, but let’s get started with the “before”.
The “Before” Story
We started by asking applied microeconomists which journals they typically submitted to before JHR. Unsurprisingly, most aim high: AEJ: Applied, AER, or QJE are common first stops. When those attempts fail (and they usually do), top fields like JHR become the next rung down. In other words, by the time a paper lands at JHR, it’s already been through one or two cycles of “maybe not for us.”
The “After” Story: Congrats on a speedy desk-rejection?
Now to the more descriptive finding. We tracked a random sample of rejected JHR manuscripts through Google Scholar and RePEc to see what happened next.
Roughly 60 percent of rejected papers eventually got published elsewhere. And here’s the kicker: that rate is almost identical whether a paper was desk-rejected or rejected after referee reports (58 percent vs 60 percent .
If peer review were a powerful sorting mechanism, you’d expect big differences between those categories. Maybe referee rejections would do much better, since at least they received feedback. But no, the fate of a rejected paper looks more like a coin toss than a merit filter. Desk rejected? You might still publish. Sent out for review? About the same odds.
One advantage of a desk-rejection is speed. Not having to wait months for feedback you might not take into account. I personally love speed and have always been desk-rejected at JHR, so I was quite happy with that result.
And what about the 40 percent that never make it? Many of these include job market papers (JMP). For those not familiar with JPMs, they are the flagship projects of early-career economists. They appear once on a CV and then vanish forever.
This is consistent with evidence from another of my papers (link). For this analysis, we searched 150 job market candidates’ CVs and webpages for information on whether their JMP is under revision (R&R) or published within a few years. About 20 percent of JMPs are published while around 11 percent are under revision. That means nearly 70 percent of job-market papers had not reached R&R or publication at all even though they’re central to hiring decisions.
Obviously this 70% will go down as some job market candidates get their JMP published 5-10 years after their PhD. But let’s be realistic, most will never publish it.
The Graveyard of the Unpublished
Finally, there’s the invisible side: all the papers that never get submitted in the first place. In our survey of applied microeconomists, about 30 percent admitted they stopped a research project or avoided submission after getting null results. That behavior alone shapes the research record. The graveyard isn’t just working papers that never publish, but studies that never even leave the hard drive.
Combine that with the job market paper numbers, and it’s hard to avoid the conclusion: the typical economics paper has a high probability of dying young, regardless of peer review status.
Advice to young economists: don’t worry too much if your paper is desk-rejected. Just submit elsewhere and thank the editor for not sending your paper for a lengthy review!