Already some weeks ago I have finished up some thoughts for a Replication Bayes factor for ANOVA contexts, which resulted in a manuscript that is available as pre-print at arXiv. The theoretical foundation was laid out before by Verhagen & Wagenmakers (2014) and my manuscript is mainly an extension of their approach. We have another paper coming up where we will use it to evaluate the success of an attempted replication of an interaction effect.

From the abstract:

With an increasing number of replications in psychological science, the question of how to evaluate the outcome of a replication attempt deserves careful consideration. Different approaches have been described before. Bayesian approaches allow to incorporate prior information into the analysis of the replication attempt by their design. The Replication Bayes Factor, introduced by Verhagen and Wagenmakers (2014), is outlined and extended to F-tests in multi-group, fixed-effect ANOVA designs. Derivation of likelihood functions and computational strategies are explained. Simulations and examples are presented to facilitate the understanding and in order to demonstrate the usefulness of this approach. The Replication Bayes Factor for F-tests is discussed in the context of evaluating replication attempts and compared to other approaches.

The Replication Bayes factor takes the posterior from an original experiment as the prior to the evaluation of the Replication attempt. While Verhagen & Wagenmakers (2014) use a Normal approximation to the posterior, I use MCMC sampling to gain samples from the true posterior of the original experiment. In the case of t-tests the differences are minor, but for F-tests the Normal approximation is way off (see picture above).

The paper is currently under review, but the preprint is already available. I will upload the R script for the computation to GitHub soon, at least when our manuscript on another replication attempt is ready (which will, of course, have pre-registration, open data, open material and open analysis).

References

• Verhagen, J., & Wagenmakers, E.-J. (2014). Bayesian tests to quantify the result of a replication attempt. Journal of Experimental Psychology: General, 143(4), 1457–1475. http://doi.org/10.1037/a0036731