the repliCATS project aim is to crowdsource predictions about the reliability of published research.
why is it important to predict replicability?
In the past 10 years, science has faced a ‘replication crisis’. Researchers have been unable to replicate the results of several landmark studies in medicine, psychology, economics and other fields, causing many to question the scientific evidence base we use to make decisions.
But, we can’t afford to test and replicate every piece of research before it’s published – it will be too costly and time consuming.
If we can accurately predict credible research, our project could transform how end-users – from academics to policy makers – can assess the reliability of social scientific research.
–– Prof Fiona Fidler, chief investigator of repliCATS project
We aim to estimate the replicability of 3,000 published social scientific research claims in: Business research, Criminology, Economics, Education, Political Science, Psychology, Public Administration, and Sociology.
The repliCATS project is part of a research program called SCORE, funded by DARPA, that eventually aims to build automated tools that can rapidly and reliably assign confidence scores to social science research claims.
The repliCATS project is led by Prof Fiona Fidler. We are a group of interdisciplinary researchers from the School of BioSciences, School of Historical and Philosophical Studies, and the Melbourne School of Engineering at the University of Melbourne, with collaboration from the Centre for Environmental Policy at Imperial College London.
We hope to develop more accurate, better-calibrated techniques to elicit estimates of the replicability of social & behavioural science evidence.
Our approach is to adapt and deploy the IDEA protocol developed here at the University of Melbourne to elicit group judgements for the likely replicability of 3,000 research claims.
The research we will undertake as part of the repliCATS project will include the largest ever empirical study on how scientists reason about other scientists’ work, and what factors makes them trust it.
We have built a custom online platform to deploy the IDEA protocol. This platform will have a life beyond the repliCATS project: it will be able to be used in the future to enhance group judgements on a wide range of topics, in a number of disciplines.