Research is messy: Two cases of pre-registrations

Pre-registrations are becoming increasingly important for studies in psychological research. This is a much needed change since part of the “replication crisis” has to do with too much flexibility in data analysis and interpretation (p-hacking, HARK’ing and the like). Pre-registering a study with planned sample size and planned analyses allows other researchers to understand what the initial thinking of the authors was, how the data fits to the initial hypothesis and where are differences between the study protocol and study results. In theory, it looks very simple: you think about a problem, you conceive a study, lay out the plan, register it, collect data, analyse and publish. 

However, in practice, research is messy: You might have problems collecting the sample size you planned or your manipulation does not work the way you thought. Or while you are collecting data, you read up on other methods or theories that might be more fitting to your study than what you knew when you pre-registered it.

In the recent months, we worked on two different studies that were pre-registered that worked out quite differently.

Our first study was a replication study. Replications are easy to pre-register because you already have the original study with all the research design and analyses given. In direct or close conceptual replications you therefore already have a good starting point.
Data collection went fine and data analysis was carried out according to the registration. We have conducted some further analysis that was not pre-registered, but those were simply for further understanding: Nothing has changed in the interpretation or the general direction of the analysis. Thus, we should have a clear match between our pre-registration and the final manuscript.

The second study was conducted outside the university for a conference. We wanted to participate and had already a clear view on the question we would like to investigate. Since the submission deadline was coming a to a close, we quickly wrote down our abstract and the pre-registration. The registration reflects our initial hypothesis and thinking very well, especially concerning the direction of the effect we expected. After the submission we started to develop the tools we needed and started the data-collection.
While collecting the data several things happened: First, we noticed that some of the variables cannot be measured as good as we thought. Second, I read more on different analytical approaches, namely Bayesian estimation, that I now think fit better to the question than what was written in the pre-registration. And third, we noticed that we also wanted to have a certain group of participants represented in our sample, so we recruited them additionally.
Some further decisions were made when we analysed the data: We dropped a variable, used a different approach and found a good interpretation of the data that, however, differed from our registered hypothesis.

Our final manuscript is now quite different to the pre-registration. The general idea is the same and our focal hypothesis is unchanged, but some differences exist nonetheless. What we will do, is to present our results but note where we changed course and why we did it. Everyone will (hopefully) be able to follow our thinking and understand why we did what we did. Others might have made other choices along the way, but this is what conference discussions are for. The key is the transparency of the whole process.

In a world without pre-registration we would have been able to write a completely different summary in the end where every single of our hypothesis was confirmed — but nobody would be able to know if this was our initial plan or solely post-hoc reasoning.1

Pre-registrations are a great way for researchers to fix their hypothesis and research design before data-collection. While this decreases the researcher’s flexibility, it allows others to clearly see the development of a research approach. A pre-registered study that fails to show some effect is worth a lot more than a low-powered, non-registered study that only reports significant results.
Pre-registrations do not stop you from additional, exploratory analysis beyond the registered study protocol. This is another recommendation for the scientific practice that has been around some time: clearly separate confirmatory and exploratory results. Pre-registrations simply make it transparent where the line between confirmation and exploration goes.

And, yes, science is messy. Your data will not be as you planned, your analysis may change, your data might tell you something you didn’t expect. And it is fine to deviate from the pre-registered study. When you write up the results, you should just clearly explain where and why you did this. Reviewers and readers will then be able to clearly see the bigger picture of your research question. They might disagree with your choices, but a discussion based on a transparent research process is much more valuable than a discussion where only one side does know which data-dependent decisions were made.

Pre-registration will generally stop HARK’ing and undocumented flexibility and data-collection and analysis. But if your research findings rely on that… well, then it might be the best time to start pre-registering today and improve your scientific practice.

And lastly, it is so easy to pre-register a study: The easiest way is to simple answer some questions for a one-pager at AsPredicted.org. Or you go “all-in” and register not only your study and analysis plan, but also any material you will use at the Open Science Framework.

References and further literature

  • Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., … Wicherts, J. M. (2013). Recommendations for Increasing Replicability in Psychology. European Journal of Personality, 27(2), 108–119. http://doi.org/10.1002/per.1919
  • Hardwicke, T. (2016). A pre-registration primer. London. Retrieved from https://osf.io/8uz2g/
    Kerr, N. L. (1998). HARKing: Hypothesizing After the Results are Known. Personality and Social Psychology Review, 2(3), 196–217. http://doi.org/10.1207/s15327957pspr0203_4
  • Franco, A., Malhotra, N., & Simonovits, G. (2016). Underreporting in Psychology Experiments: Evidence From a Study Registry. Social Psychological and Personality Science, 7(1), 8–12. http://doi.org/10.1177/1948550615598377
  • Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. http://doi.org/10.1177/0956797611417632
  • Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2012). A 21 Word Solution. SSRN Electronic Journal, 1–4. http://doi.org/10.2139/ssrn.2160588
  • Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., … van ’t Veer, A. (2014). The Replication Recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50(1), 217–224. http://doi.org/10.1016/j.jesp.2013.10.005
  1. In hindsight we might have written a better pre-registration if we would have taken more time to plan the study and develop the tools before we pre-register. That’s our learning from this study…

One thought on “Research is messy: Two cases of pre-registrations”

Leave a Reply

Your email address will not be published. Required fields are marked *