Replication studies in economics🍋
Many journals are reluctant to publish comments & replications. Here's where to publish them
Replication studies play a crucial role in economics by ensuring the reliability, validity, and robustness of research findings. In an era where policy decisions and societal interventions heavily rely on economic research, the ability to replicate and validate research findings is important for making informed decisions and advancing knowledge.
A disturbing failure to replicate
Replications in economics became more mainstream after the influential Science paper by Colin Camerer, Anna Dreber, and others was published in 2016.1 This paper replicated 18 studies from the AER2 and QJE published between 2011 and 2014. In just 61% (11 out of 18) of those studies, the replication found a significant effect in the same direction as in the original study.
Replication studies serve as a cornerstone of scientific integrity and transparency. Economics, like other empirical sciences, relies on data-driven analysis to draw conclusions about complex economic phenomena. However, the validity of these conclusions can often be challenged due to various factors such as data limitations, methodological choices, or even unintentional errors. By replicating papers, researchers can verify whether the original findings hold under different conditions, datasets, or methodologies. This process not only enhances the credibility of the research but also fosters trust.
Replications contribute to the cumulative nature of knowledge. Many economic theories and empirical results build upon earlier findings. However, the reliability of these findings can sometimes be uncertain, especially when they are based on limited samples or specific contexts.
Fostering quality assurance
Replication studies allow researchers to confirm whether the findings are generalizable across different populations, time periods, or geographical regions. This cumulative process helps in refining theories, identifying boundaries, and uncovering inconsistencies that may require further investigation.
Additionally, replication studies encourage openness and collaboration within the economics community. They promote constructive dialogue among researchers about the strengths and limitations of different approaches and methodologies. By openly discussing replication efforts and their outcomes, we can collectively enhance research practices, improve data transparency, and foster a culture of quality assurance in economic research.
Introducing the Journal of Comments and Replications in Economics
While some journals publish replication studies (e.g. Journal of Applied Econometrics, JPE: Micro, JESA, and Empirical Economics), the new Journal of Comments and Replications in Economics (JCRE) aims to be the premier outlet for articles that comment on or replicate previously published articles.
Because many journals are reluctant to publish comments and replications, JCRE was founded to provide an outlet for research that explores whether published results are correct, robust, and/or generalizable and to publish replications defined as any study that directly addresses the reliability of a specific claim from a previously published study.
The editorial board of JCRE consists of W. Robert Reed (University of Canterbury), Marianne Saam (University of Hamburg), and myself (Dennis Wesselbaum, University of Otago). We are supported as editors by the Leibniz Information Centre for Economics (ZBW) and the German Research Foundation (DFG).
Our advisory board consists of the following outstanding scholars: David Autor (MIT), Anna Dreber Almenberg (Stockholm School of Economics), Richard Easterlin (USC), Edward Leamer (UCLA), David Roodman (Open Philanthropy), and Jeffrey Wooldridge (MSU).
Replication studies are indispensable
In conclusion, replication studies are indispensable in economics for compelling reasons. These studies:
uphold scientific rigor and transparency,
contribute to the cumulative advancement of knowledge,
help identify biases and flaws in research, and
promote openness and collaboration among researchers.
With the increase in the availability of large data sets, the advancements in computer power, and the development of new econometric tools, the importance of replicating and validating research findings will only grow. To this end, we invite you to submit your replication studies to the JCRE and to promote the journal in your networks and among your students.
Replication studies serve as a cornerstone of scientific integrity and transparency.
Camerer et al. (2016). Evaluating replicability of laboratory experiments in economics. Science 10.1126/science.aaf0918 (2016).
Acronym decoder:
AER = American Economic Review
QJE = Quarterly Journal of Economics
JPE = Journal of Political Economy
JESA = Journal of the Economic Science Association
MIT = Massachusetts Institute of Technology
USC = University of Southern California
UCLA = University of California, Los Angeles
MSU = Michigan State University