questionable research practices

p-hacking destroys everything (not only p-values)

In the context of problems with replicability in psychology and other empirical fields, statistical significance testing and p-values have received a lot of criticism. And without question: much of the criticism has its merits. There certainly are problems with how significance tests are used and p-values are interpreted.¹

However, when we are talking about “p-hacking”, I feel that the blame is unfairly on p-values and significance testing alone without acknowledging the general consequences of such behaviour in the analysis.² In short: selective reporting of measures and cases³ invalidates any statistical method for inference. When I only selectively report variables and studies, it doesn’t matter whether I use p-values or Bayes factors — both results will be useless in practice.

Already in September last year, Der Spiegel published an interview with Peter Wilmshurst, a British medical doctor and whistleblower who made fraudulent practices in medical research public:

In the course of the 66-year-old’s career, he conducted studies for pharmaceutical and medical devices companies, and unlike many of his colleagues, never hesitated to publish negative results. He’s been the subject of multiple cases of legal action and risked bankruptcy and his reputation to expose misconduct in the pharmaceutical industry.

A very interesting article that’s worth reading. Fact is, that companies who have a strong economic interest in the scientific process will have an impact on the quality of the research. It is, again and again, horrible to learn how far companies try to go – and often successfully do. While medical companies has always been an obvious target (and perpetrator), the problem runs deeper than the narrative of “Big Pharma”.