Promoting Reproducibility and Replicability in Political Science
Our chair, Abel Brodeur, Kevin Esterling and many collaborators just published a piece in Research & Politics! The piece is entitled "Promoting reproducibility and replicability in political science".
The article reviews and summarizes current reproduction and replication practices in political science. First, it reviews data availability policies for 28 leading political science journals. The analysis was conducted in early June 2023. They report that "four of the journals have dedicated data/replication editors, namely, Journal of Politics, Political Analysis, Political Communication, Political Science Research and Methods, and Quarterly Journal of Political Science. The American Journal of Political Science also has a verification process carried out by the Odum Institute for Research in Social Science at the University of North Carolina at Chapel Hill. Just one of the journals has no information on its website with regard to data or codes—Comparative Politics. Of the remaining 27 journals that were sampled, six encourage the sharing of data/codes and 21 mandate it." See Table 1 in the article.
They also present the results from a survey of editors about their willingness to publish comments and replications. The survey responses are presented below and summarized in their Figure 1.
They discuss new initiatives that seek to promote and generate high-quality reproductions and replications. Finally, they make three recommendations: "(1) We call on the American Political Science Association sections (or at least the experiments section) to create an award for best replication/extension study. (2) We urge the creation of an outlet dedicated to replications backed up by one of the large disciplinary professional associations (for which impact factor might not be a primary consideration). (3) We also recommend that more political science journals start using data editors to improve computational reproducibility."
Table 1. Data availability policy and data editor.
| Journal | Data availability policy? | Data editor? |
|---|---|---|
| American journal of political science | Yes | Independent |
| American political science review | Yes | No |
| American politics research | Encourage | No |
| British journal of political science | Yes | No |
| Comparative political studies | Yes | No |
| Comparative politics | No | No |
| European journal of political research | Yes | No |
| International organization | Yes | No |
| International studies quarterly | Yes | No |
| Journal of conflict resolution | Yes | No |
| Journal of experimental political science | Yes | No |
| Journal of law, economics, and organization | Yes | No |
| Journal of politics | Yes | Yes |
| J. Of public administration research & theory | Yes | No |
| Journal of theoretical politics | Encourage | No |
| Legislative studies quarterly | Encourage | No |
| Political analysis | Yes | Yes |
| Political behavior | Yes | No |
| Political communication | Yes | Yes |
| Political geography | Encourage | No |
| Political psychology | Encourage | No |
| Political research quarterly | Yes | No |
| Political science research and methods | Yes | Yes |
| Public choice | Encourage | No |
| Public opinion quarterly | Yes | No |
| Quarterly journal of political science | Yes | Yes |
| Research & politics | Yes | No |
| Review of international organizations | Yes | No |
Survey
We also conducted a short survey of journal editors. We share the answers from this survey below. The survey included the following text followed by two questions: "We're writing to you in regard to your role as editors of the [journal name]. We are doing research on post-publication review through replications and reanalyses on previously published papers. In some (but not all) journals the format we are talking about are published as comments. A comment directly addresses and challenges a previously published paper. As editors of the [journal name], we would love to hear how your journal handles the publication of comments on previously published papers. We would really appreciate if you could briefly answer the following two questions. We are fully aware that these are rather generic questions, and it is hard to respond with an unconditional yes or no. So, feel free to elaborate briefly if you like."
(1) Do you publish comments in the [journal name]? By comment we mean a paper that discusses and potentially challenges the empirical results from another paper, for example based on a reanalysis or additional robustness checks.
(2) If yes, do you only publish comments on original papers that have previously been published in the [journal name] or do you also publish comments on original papers that have been published elsewhere?
| Discipline | Journal | Editor Answered | Q1 - Code | Q2 - Code | Long Answer |
|---|---|---|---|---|---|
| Political Science | American Journal of Political Science | Kathleen Dolan and Jennifer Lawless | No | No | We don’t publish comments. |
| Political Science | American Political Science Review | Aili Mari Tripp | Yes | See long answer. | Q1: Yes, we would consider publishing a comment in our Letter (research notes) format that challenges the empirical results from another paper if... 1) the article is of interest to a sufficiently broad section of our readership; or 2) if the article makes a methodological advance that would be of interest to a broader readership. Q2: We have generally avoided publishing comments on manuscripts published elsewhere. We privilege commentary on our own articles, as we believe that the responses to articles in other journals belong in those journals. But this is not to say that we would never consider doing this. It would depend on the importance of the debate and the extent to which the issues are of general interest. But as a general principle, we would not do so. |
| Political Science | British Journal of Political Science | Robert A. Johns | Yes | Yes | Q1: Yes, we do. Not many -- perhaps one every second issue, or something like that -- but some... (Quite a few -- maybe even half -- of these are accompanied by a follow-up rejoinder from the original author, although that's as far as we take an exchange.) Q2: We don't have a rule against comments on papers published elsewhere, and occasionally that happens, but the clear majority of our comments are on BJPolS papers. That's partly because we feel more of an obligation to correct/refine findings published in our own journal. It's also because I guess there's a natural 'level' issue here: we probably wouldn't publish a comment on an article in a journal well below us in the rankings/prestige/hierarchy, simply because the original contribution is unlikely to have been of a size that we'd have published in the first place; meanwhile, someone wanting to comment on an article in the AJPS is unlikely to want to publish it with us. |
| Political Science | Comparative Political Studies | David Samuels | No | No | We have not published comments and have no plans to do so at present. |
| Political Science | Comparative Politics | Yekaterina Oziashvili | No | No | We do not publish comments in Comparative Politics. |
| Political Science | European Journal of Political Research | Emiliano Grossman | See long answer. | The quick answer ist that we do not have a specific format for this... We have done it recently in an ad hoc move in response to an author who approached us regarding a recently published paper (by us). We published his comment as a "research note", an existing format. While we may do this again, subject to agreement bw the three editors, we do not currently have a policy on this. |
|
| Political Science | International Organization | Erik Voeten | Yes | Yes | We do not have a separate comments or letters to the editors section... We do publish research notes that are based on replications of articles we have published and occasionally articles that were published elsewhere. Those notes go through the regular review process. |
| Political Science | International Studies Quarterly | Brandon C. Prins | Yes | No | Q1. Yes. Q2: Only publish responses to manuscripts published in ISQ. |
| Political Science | Journal of Conflict Resolution | Paul Huth | Yes | No | For Q1 the answer is that JCR occasionally publishes what you term comments... For Q2 when JCR publishes comments they are in relation to a prior JCR publication. |
| Political Science | Journal of Experimental Political Science | Kevin Arceneaux | Yes | See long answer. | This is a bit of a theoretical question for us, because we have yet to receive a submission directly challenging something published in the journal,... , but if we did, we would be open to publishing it as a comment that allows the authors of the original piece to respond. For your second question, I have a complicated response. We would only consider publishing a paper that “comments” on a publication in a venue other than JEPS if it consisted of original research (e.g., a replication study), but we would not consider it if it only re-analyzed data published elsewhere. I should also add that these polices are not written in stone and perhaps a future editorial team will do something different! |
| Political Science | Journal of Public Administration Research and Theory | Mary K. Feeney | No | No | We do not do comments. |
| Political Science | Legislative Studies Quarterly | Guillermo Rosas | Yes | Yes | The answers to your questions are pretty straightforward in LSQ's case... We do not have an established policy on publishing comments. We would not discourage submissions that seek to discuss or challenge the empirical results from published papers, either in LSQ or elsewhere, but we have not sought in the past to invite such comments. However, the idea strikes me as worth following up. |
| Political Science | Political Analysis | Jeff Gill | No | No | No, We require replication of every analysis before publication. Our policies are at our submissions page online. |
| Political Science | Political Communications | Regina G. Lawrence | No | No | We do not publish comments of the sort you describe. |
| Political Science | Political Research Quarterly | Seth C. McKee | No | No | As a rule, no, we do not publish comments in Political Research Quarterly... We have not done this at all under our editorial tenure. |
| Political Science | Public Opinion Quarterly | Eric Plutzer | Yes | Yes | Q1: Yes we do. But we do not distinguish these from any other submission... Indeed, every submission “discusses” empirical results from other papers and a large majority of submissions “challenge” prior empirical results. A “Comment” is often a special case, one that focuses on a single paper’s conclusions. If we receive such a submission, we treat it like any other, with one exception: at least one author of the targeted article is invited to serve as a reviewer in the double-blind condition. If the commenter is assured that one of the original authors will be a reviewer, I think there is a pressure towards professional, less aggressive and less pejorative framing. The original author can choose to disclose their identity or not. We can take into account potential conflicts of interest when reviewing the full set of reviews. Here is a recent example: Urbatsch, R., 2020. Revisiting “The First-Daughter Effect”. Public Opinion Quarterly, 84(2), pp.523-537. I think the process worked well in this case. Q2: This is irrelevant since we do not treat comments differently from any other submission. If the contribution to knowledge is substantial, it makes no difference where the original findings were published (and vice verse). It’s useful to add that if a third party points out a simple error (e.g., a variable was reverse coded and thereby mis-interpreted by the authors, or the authors incorrectly included or excluded cases, etc.), the preferred mechanism is collegial: (a) the person who identified the error contacts the authors, hopefully copying us in the email thread, and then (b) the authors formulate a corrigendum , (c) which we publish. There is no need for the “corrector” to have authorship in these cases. We publish corrigenda from time to time and many are the result of these normal collegial interactions. We’ve never had a retraction, but in the case of a more serious error, that could occur. That is how science ought to work. |
| Political Science | Quarterly Journal of Political Science | Anthony Fowler | Yes | Yes | The answer to both questions is yes... We do not formally distinguish between original articles and research comments unless the author would like to explicitly classify their article as a research note. We are certainly open to submissions that are largely comments on or reassessments of previously published articles. Just like all submissions, we would evaluate them on the basis of whether they are high-quality, novel, and make a significant contribution to our field. If there is an opportunity for authors to make a valuable contribution beyond their commentary on one specific paper, we would encourage them to do so, but we are open to publishing papers that are primarily responses to or critiques of other studies. |
| Political Science | Research & Politics | Kevin M. Esterling | Yes | Yes | Yes to both questions. R&P does strongly encourage and publish both replications and comments... We have a clear policy for dealing with replication studies which is detailed here: https://journals.sagepub.com/author-instructions/RAP#ArticleTypes. |
| Political Science | Review of International Organizations | Axel Dreher | Yes | Yes | Q1: Yes. Q2: Also elsewhere (explicit journal policy). |
See the article for more details.
Figure 1
