1. Home
  2. Volume 7 - Number 2
  3. A partial solution for the replication crisis in economics
New submission
Issues
Journal Information
Aims and ScopeEditorial BoardEditorial Advisory BoardJournal Policies
Resources
For Readers
For Authors
For Editors
For Reviewers
Publication EthicsCall for Special Issue
ISSN
ISSN 2615-9821
ISSN(Previous) 2588-1396

Volume 7 - Number 2 | July

A partial solution for the replication crisis in economics

William M. Briggs

12/07/2023 55 128 0
Like Buy now 20.000 đ
Bayes factors Hypothesis testing Predictive modeling Replication crisis.

Abstract:

Purpose
Important research once thought unassailable has failed to replicate. Not just in economics, but in all science. The problem is therefore not in dispute nor are some of the causes, like low power, selective reporting, the file drawer effect, publicly unavailable data and so forth. Some partially worthy solutions have already been offered, like pre-registering hypotheses and data analysis plans.

Design/methodology/approach
This is a review paper on the replication crisis, which is by now very well known.

Findings
This study offers another partial solution, which is to remind researchers that correlation does not logically imply causation. The effect of this reminder is to eschew “significance” testing, whether in frequentist or Bayesian form (like Bayes factors) and to report models in predictive form, so that anybody can check the veracity of any model. In effect, all papers could undergo replication testing.

Originality/value
The author argues that this, or any solution, will never eliminate all errors.

References:

  1. Baker, M. (2016), “1,500 scientists lift the lid on reproducibility”, Nature, Vol. 533, pp. 452-454.
  2. Banerjee, A., Duflo, E., Finkelstein, A., Katz, L.F., Olken, B.A. and Sautmann, A. (2020), In Praise of Moderation: Suggestions for the Scope and Use of Pre-analysis Plans for Rcts in Economics, Working Paper 26993, National Bureau of Economic Research.
  3. Bernardo, J.M. and Smith, A.F.M. (2000), Bayesian Theory, Wiley, New York.
  4. Bose, S. (2004), “On the robustness of the predictive distribution for sampling from finite populations”, Statistics and Probability Letters, Vol. 69 No. 1, pp. 21-27.
  5. Briggs, W.M. (2016), Uncertainty: The Soul of Probability, Modeling and Statistics, Springer, New York.
  6. Briggs, W.M. (2019), “Reality-based probability and statistics: solving the evidential crisis”, Asian Journal of Business and Economics, Vol. 3 No. 1, pp. 37-80.
  7. Briggs, W.M. and Nguyen, H.T. (2019), “Clarifying asa’s views on p values in hypothesis testing”, Asian Journal of Business and Economics, Vol. 3 No. 2, pp. 1-16.
  8. Briggs, W.M. (2019), “Everything wrong with p values under one roof”, in Kreinovich, V., Thach, N.N., Trung, N.D. and Thanh, D.V. (Eds), Beyond Traditional Probabilistic Methods in Economics, Springer, New York, pp. 22-44.
  9. Bruns, S.B. and Ioannidis, J.P.A. (2016), “p-curve and p-hacking in observational research”, PLoS ONE, Vol. 11 No. 2, e0149144.
  10. Camerer, C.F., Dreber, A., Forsell, E., Ho, T.-H., Huberand, J., Johannesson, M., Kirchler, M., Almenberg, J. and Altmejd, A. (2016), “Evaluating replicability of laboratory experiments in economics”, Science, Vol. 351 No. 6280, pp. 1433-1436.
  11. Camerer, C.F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B.A., Pfeiffer, T., Altmejd, A., Buttrick, N., Chan, T., Chen, Y., Forsell, E., Gampa, A., Heikensten, E., Hummer, L., Imai, T., Isaksson, S., Manfredi, D., Rose, J., Wagenmakers, E.-J. and Wu, H. (2018), “Evaluating the replicability of social science experiments in nature and science between 2010 and 2015”, Nature Human Behaviour, Vol. 2 No. 9, pp. 637-644.
  12. Charlton, A. (2023), “Replications of marketing studiesass”, available at: https://openmkt.org/research/replications-of-marketing-studies/
  13. Clarke, B.S. and Clarke, J.L. (2018), Predictive Statistics, Cambridge University Press, Cambridge.
  14. Fanelli, D. (2017), “Is science really facing a reproducibility crisis, and do we need it to?”, PNAS, Vol. 115 No. 11, pp. 2628-2631.
  15. Hájek, A. (1996), “Mises redux—redux: fifteen arguments against finite frequentism”, Erkenntnis, Vol. 45 Nos 2-3, pp. 209-227.
  16. Harvey, C.R. (2019), “Replication in financial economics”, SSRN, available at SSRN: https://ssrn.com/abstract=3409466
  17. Horton, R. (2015), “Offline: what is medicine's 5 sigma?”, The Lancet, Vol. 385 No. 9976, p. 1380.
  18. Hossenfelder, S. (2017), “Science needs reason to be trusted”, Nature Physics, Vol. 13 No. 4, pp. 316-317.
  19. Ioannidis, J.P.A. (2005), “Contradicted and initially stronger effects in highly cited clinical research”, JAMA, Vol. 294 No. 2, pp. 218-228.
  20. Klein, R.A., Vianello, M., Hasselman, F., Adams, B.G., Adams, R.B. Jr, Alper, S., Aveyard, M., Axt, J.R., Babalola, M.T., Bahník, Š., Batra, R., Berkics, M., Bernstein, M.J., Berry, D.R., Bialobrzeska, O., Binan, E.D., Bocian, K., Brandt, M.J., Busching, R., Rédei, A.C., Cai, H., Cambier, F., Cantarero, K., Carmichael, C.L., Ceric, F., Chandler, J., Chang, J.-H., Chatard, A., Chen, E.E., Cheong, W., Cicero, D.C., Coen, S., Coleman, J.A., Collisson, B., Conway, M.A., Corker, K.S., Curran, P.G., Cushman, F., Dagona, Z.K., Dalgar, I., Rosa, A.D., Davis, W.E., de Bruijn, M., De Schutter, L., Devos, T., de Vries, M., Doğulu, C., Dozo, N., Dukes, K.N., Dunham, Y., Durrheim, K., Ebersole, C.R., Edlund, J.E., Eller, A., English, A.S., Finck, C., Frankowska, N., Freyre, M.Á., Friedman, M., Galliani, E.M., Gandi, J.C., Ghoshal, T., Giessner, S.R., Gill, T., Gnambs, T., Gómez, Á., González, R., Graham, J., Grahe, J.E., Grahek, I., Green, E.G.T., Hai, K., Haigh, M., Haines, E.L., Hall, M.P., Heffernan, M.E., Hicks, J.A., Houdek, P., Huntsinger, J.R., Huynh, H.P., Ijzerman, H., Inbar, Y., Innes-Ker, Å.H., Jiménez-Leal, W., John, M.-S., Joy-Gaba, J.A., Kamiloğlu, R.G., Kappes, H.B., Karabati, S., Karick, H., Keller, V.N., Kende, A., Kervyn, N., Knežević, G., Kovacs, C., Krueger, L.E., Kurapov, G., Kurtz, J., Lakens, D., Lazarević, L.B., Levitan, C.A., Lewis, N.A., Jr, Lins, S., Lipsey, N.P., Losee, J.E., Maassen, E., Maitner, A.T., Malingumu, W., Mallett, R.K., Marotta, S.A., Mededovic, J., Mena-Pacheco, F., Milfont, T.L., Morris, W.L., Murphy, S.C., Myachykov, A., Neave, N., Neijenhuijs, K., Nelson, A.J., Neto, F., Nichols, A.L., Ocampo, A., O'Donnell, S.L., Oikawa, H., Oikawa, M., Ong, E., Orosz, G., Osowiecka, M., Packard, G., Pérez-Sánchez, R., Petrović, B., Pilati, R., Pinter, B., Podesta, L., Pogge, G., Pollmann, M.M.H., Rutchick, A.M., Saavedra, P., Saeri, A.K., Salomon, E., Schmidt, K., Schönbrodt, F.D., Sekerdej, M.B., Sirlopú, D., Skorinko, J.L.M., Smith, M.A., Smith-Castro, V., Smolders, K.C.H.J., Sobkow, A., Sowden, W., Spachtholz, P., Srivastava, M., Steiner, T.G., Stouten, J., Street, C.N.H., Sundfelt, O.K., Szeto, S., Szumowska, E., Tang, A.C.W., Tanzer, N., Tear, M.J., Theriault, J., Thomae, M., Torres, D., Traczyk, J., Tybur, J.M., Ujhelyi, A., van Aert, R.C.M., van Assen, M.A.L.M., van der Hulst, M., van Lange, P.A.M., van ’t Veer, A.E., Vásquez-Echeverría, A., Vaughn, L.A., Vázquez, A., Vega, L.D., Verniers, C., Verschoor, M., Voermans, I.P.J., Vranka, M.A., Welch, C., Wichman, A.L., Williams, L.A., Wood, M., Woodzicka, J.A., Wronska, M.K., Young, L., Zelenski, J.M., Zhijia, Z. and Nosek, B.A. (2018), “Many labs 2: investigating variation in replicability across samples and settings”, Advances in Methods and Practices in Psychological Science, Vol. 1 No. 4, pp. 443-490.
  21. Knuteson, B. (2016), “The solution to science's replication crisis”.
  22. Lancaster, T. (2004), An Introduction to Modern Bayesian Econometrics, Blackwell Publishing, New York.
  23. Macleod, M. and the University of Edinburgh Research Strategy Group (2022), “Improving the reproducibility and integrity of research: what can different stakeholders contribute?”, BMC Research Notes, Vol. 15 No. 1, p. 146.
  24. Mede, N.G., Schäfer, M.S., Ziegler, R. and Weißkopf, M. (2021), “The “replication crisis” in the public eye: Germans' awareness and perceptions of the (ir)reproducibility of scientific research”, Public Understanding of Science, Vol. 30 No. 1, pp. 91-102, PMID: 32924865.
  25. Miyakawa, T. (2020), “No raw data, no science: another possible source of the reproducibility crisis”, Molecular Brain, Vol. 13, pp. 1-6.
  26. Page, L., Noussair, C.N. and Slonim, R. (2021), “The replication crisis, the rise of new research practices and what it means for experimental economics”, Journal of the Economic Science Association, Vol. 7 No. 2, pp. 210-225.
  27. Reed, W. (2017), “Replication in labor economics”, IZA World of Labor, Vol. 413.
  28. Sharpe, D. and Poets, S. (2020), “Meta-analysis as a response to the replication crisis”, Psychologie canadienne, Vol. 61 No. 4, pp. 377-387.
  29. Smaldino, P.E. and McElreath, R. (2016), “The natural selection of bad science”, Royal Society Open Science, Vol. 3 No. 9, doi: 10.1098/rsos.160384.
  30. Smith, R.L. (1998), “Bayesian and frequentist approaches to parametric predictive inference (with discussion)”, Bayesian Statistics, Oxford University Press, Vol. 6, pp. 589-612.
  31. Spanos, A. (2022), “Frequentist model-based statistical induction and the replication crisis”, Journal of Quantitative Economics, Vol. 20 No. 1, pp. 133-159.
  32. Trafimow, D. (2018), “An a priori solution to the replication crisis”, Philosophical Psychology, Vol. 31 No. 8, pp. 1188-1214.
  33. Williams, C.R. (2019), “How redefining statistical significance can worsen the replication crisis”, Economics Letters, Vol. 181, pp. 65-69.

 

Related articles