This term, the Durham Research Methods Café is holding a series of cross-disciplinary conversations concerning topics fundamental to research. The first kicked off the series with a fascinating discussion on the replication crisis, involving staff and postgraduate students from departments across the university. Particularly an issue in the social sciences, the replication crisis refers to the issue of many studies being difficult to reproduce. As an example, we discussed the Reproducibility Project, where only 39 of 100 psychology studies were replicated, throwing into question the reliability of research. But what is causing the replication crisis? We soon identified three separate stages of research where issues could arise: data collection, data analysis, and the interpretation of results.
First, we considered data collection, particularly sampling methods. Although only 12% of the world’s population are WEIRD (Western, educated, industrialised, rich, and democratic), this demographic represents up to 80% of study participants. Therefore, we must consider how far we can generalise research findings; if our study populations only represent a small demographic, how could findings be replicated outside of the setting in which that particular research project is conducted? We discussed the book Invisible Women by Caroline Criado Perez, in which the example of medical trials is used. When medical trials are conducted exclusively on men, how can we be certain that prescribing the outcomes to men and women alike is a decision based on reliable science? This highlights the relevance and importance of the replication crisis.
We went on to consider issues with data analysis and statistical reproducibility. Much quantitative research relies on p values; if a hypothesis test is shown to be significant, the results tend to be taken at face value, and are more likely to be published. However, statistical reproducibility is shown to be low, with p value results which are on the boundary of significance shown to only have around a 50% chance of replication. In response to this issue, we discussed the importance of effects sizes, which allow us examine the size of the difference between two groups, rather than over-reliance on p values and ‘significant’ differences between groups.
Finally, we contemplated issues surrounding the interpretation of research. The need to emphasise the distinction between exploratory research and hypothesis testing was highlighted, in that significant results from exploratory research require further testing before conclusions can be confidently drawn. However, due to publication bias and the media, significant results are often over-emphasised and headline stories often misrepresent what has actually been found. We suggested that high impact journals should consider calls for replications and meta-analysis, in an attempt to move away from publication bias and the misrepresentation of results.
But throughout the conversation, one theme was repeated - is this actually a crisis, or just a challenge that researchers face? Phenomena naturally change over time and context, and there is value in analysing variation. We must carefully think about what exactly a replication of any study would tell us, through considering our methods of sampling, data collection, and analysis. If a study can’t be replicated, this does not necessarily mean it is not valuable, but more thought must be given as to the circumstances behind the research.
To reflect on the conversation, we considered how to move through the replication crisis as researchers. The cross-disciplinary nature of the conversation led to the discussion of learning from other disciplines, with examples from computer science of good research practices regarding sharing data and code, and examples from the social sciences regarding the practicality of replicating large-scale studies. The role of open access and transparency within research was emphasised across departments, with the feeling that we must hold ourselves to high standards within our own research practices and as we teach the next generation of researchers.
Written by Catherine Marley (PhD Student in Anthropology, funded by DRMC)
Comments
Post a Comment