Known as the Many Labs 2 project, the study is the latest of six major replication projects in the social and behavioural sciences published since 2014. The cumulative evidence suggests there is room to improve the reproducibility of findings, says Associate Professor Taciano Milfont from Victoria University of Wellington’s Centre for Applied Cross-Cultural Research in the School of Psychology.
The results of the study, contributed by 186 researchers in 36 countries, are published today in Advances in Methods and Practices in Psychological Science. The researchers were able to replicate the results of 14 of the 28 findings successfully, and to evaluate factors inhibiting replication.
Associate Professor Milfont says the study is important because prominent failures to replicate findings in psychology and other scientific disciplines have increased concerns that the published literature is not as reproducible as expected.
He says each of the 28 findings was repeated in more than 60 laboratories all over the world, resulting in a median sample size of 7021 participants. This was more than 62 times larger than the median sample size of 112 of the original studies.
One of the findings reliably replicated across the laboratories was the so-called ‘correspondence bias’, or people’s tendency to attribute behaviour to characteristics of the person rather than characteristics of the situation. In contrast, another finding suggesting that climate change beliefs could be influenced by immediately available temperature information, with stronger beliefs observed after people are subtly reminded of heat-related worlds, did not replicate.
With samples from all six populated continents, the study tested a popular argument that some psychology findings fail to replicate because the original and replication samples are different. “We were surprised that the diversity in our samples from around the world did not result in substantial diversity in the findings that we observed,” said Rick Klein, one of the project leaders, who was a visiting scholar at the University’s Centre for Applied Cross-Cultural Research in 2015 and is now a postdoctoral associate at the University of Grenoble Alpes in France.
Associate Professor Milfont says the Many Labs 2 project embodies many reforms that are spreading across disciplines to improve transparency, rigour and reproducibility of scientific findings, and to increase public access and reusability of scientific data and literature.
This includes researchers pre-registering their studies, working alongside journals to get peer review and publication acceptance prior to data collection, and the open sharing of all data, materials, and code.
“Our School of Psychology is leading the way in Open Science in New Zealand, a movement to increase openness, integrity, and reproducibility of scientific research,” said Associate Professor Milfont.
He was the only Many Labs 2 researcher collecting data in New Zealand, and his lab was one of only 15 laboratories that collected data for all experiments included in the project. But he says other academics at the School of Psychology are involved in replication projects or other Open Science initiatives in their teaching and research, and the incoming Head of School, Professor Louis Moses, who joins the University in January 2019, is one of the editors of Collabra: Psychology, the official journal of the Society for the Improvement of Psychological Science.