In December I already blogged about the ReplicationBF package, I made available on GitHub. It allows you to calculate Replication Bayes Factors for t- and F-tests. The preprint detailing the formulas for the latter was outdated and the method in the package was not optimal, so I recently updated both.
The Replication Bayes Factor was originally outlined by Verhagen & Wagenmakers (2014). The general idea is to devise a Bayes factor, that allows to quantify the statistical evidence from a replication study under consideration of the data from the original study.
There are generally many different ways how to do this. The Replication Bayes Factor has the advantage, that it can be calculated using only the reported test-statistics. Thus, only little assumptions about priors are introduced.
In my preprint, now updated on arXiv, I outline one extension and one modification to the original proposal. First, it is extended to F-tests in balanced, fixed-effect ANOVAs when only the reported F-statistic is available (no change since my first preprint). Second, the accuracy of the numerical Bayes factor approximation is improved by using Importance Sampling (newly added). As shown in the paper, in some extreme cases this makes a difference, even if general conclusions are rarely effected.
For the F-test, the Replication Bayes Factor is insensitive to the pattern and direction of the effects in an ANOVA. Therefore, the pattern needs to be inspected manually to not be fooled if the numerical value is highly in favor of a successful replication (see Example 3 in the preprint). Another way would be to include additional constraints in the model resembling the pattern of effects found in the original study. Might be an interesting starting point for future extensions, but would change the general model of the Replication Bayes Factor.
tl;dr:
- The ReplicationBF package for R allows easy calculation of Replication Bayes Factors (using Importance Sampling).
- The updated preprint details the method, shows simulations, and provides examples.