We aim to crowdsource expert predictions on the likelihood that 3,000 published social scientific research claims will replicate.
Working in local and/or virtual groups, experts will be asked to make structured judgements about the credibility of a research claim. Our method – the IDEA protocol – harnesses the power of structured group discussion in predicting likely replicability.
Why is our work important?
In the past 10 years, science has faced a ‘replication crisis’. Researchers have been unable to replicate the results of several landmark studies in medicine, psychology, economics and other fields, causing many to question the scientific evidence base we use to make decisions.
The repliCATS project could ultimately transform how users of social scientific research – from academics to policy makers – can assess the reliability of social scientific research.
The repliCATS project is part of a research program called SCORE, funded by DARPA, that aims to build automated tools to rapidly and reliably assign confidence scores to social science research claims.
SCORE will cover 8 main areas of social and behavioural science: Business research, Criminology, Economics, Education, Political Science, Psychology, Public Administration, and Sociology.
The repliCATS project is led by A/Prof Fiona Fidler. We are a group of interdisciplinary researchers from the School of BioSciences, School of Historical and Philosophical Studies, and the Melbourne School of Engineering at the University of Melbourne, with collaboration from the Centre for Environmental Policy at Imperial College London.
Our approach is to adapt and deploy the IDEA protocol developed here at the University of Melbourne to elicit group judgements for the likely replicability of 3,000 research claims.
The research we will undertake as part of the repliCATS project will include the largest ever empirical study on how scientists reason about other scientists’ work, and what factors makes them trust it.
We are building a custom online platform to deploy the IDEA protocol. This platform will have a life beyond the repliCATS project: it will be able to be used in the future to enhance expert group judgements on a wide range of topics, in a number of disciplines.
The repliCATS project is a part of A/Prof Fiona Fidler’s Interdisciplinary MetaResearch Group (IMeRG). Here is our team structure, and you can check out our team bios below.
- leadership team
Fiona is the chief investigator for the repliCATS project. She holds an Australian Research Council Future Fellowship for investigating Reproducibility and Open Science, and holds a joint appointment in History and Philosophy of Science and BioSciences at the University of Melbourne. Fiona sits on the Center for Open Science’s Open Science Badge Evaluation Committee, and is an Open Science Framework Ambassador. From 2006-2012 she was an Australian Research Council Postdoctoral Fellow with Professor Geoff Cumming in the School of Psychological Science at La Trobe University. Together, Fiona and Geoff established the field of Statistical Cognition; their joint work has motivated policy change at several journals, in both psychology and ecology. Prior to that, her PhD in Philosophy of Science examined statistical reform in psychology, medicine and ecology, the damage caused by Null Hypothesis Significance Testing, and cognitive and social resistance to methodological change.
Hannah is the research program coordinator for the repliCATS project. She is a research fellow at the University of Melbourne working in Fiona Fidler’s meta-research lab, IMeRG. She is lead author of Questionable Research Practices in Ecology and Evolution (Fraser et al. 2018), which has received widespread attention (preprint downloaded 679 times). During her PhD, Hannah also gained expert elicitation experience.
Ross Pearson is a digital supply chain transformation leader that has been the technical lead for telecommunications and mining transformations. As a delivery specialist, Ross ensures that large projects and transformation implementations are realised. In addition to his supply chain experience, Ross has worked on numerous University research projects and is currently working with Melbourne University to ensure the success on the SCORE project. Ross has recently completed an honours in Computer Science with a focus on Artificial Intelligence.
- reproducibility team
Peter Vesk (Principal Investigator)
Peter is an Associate Professor and Reader in the School of BioSciences at University of Melbourne. He has a long history of working on generalization and reliability of scientific knowledge before it was known as reproducibility, starting in plant ecology. He is an Associate Editor at Journal of Ecology, the most highly ranked journal in plant ecology. As a founding editor of Hot Topics in Ecology, designed to provide evidence based statements on ecological topics relevant to policy and management, he is keenly interested in participatory methods of providing reliable scientific knowledge. He has > 100 journal articles, with >4700 citations and H=36 (Scopus). Vesk’s research focus is gathering, formalizing and generalizing knowledge for management. This entails attention to methodology of data collection, use and model evaluation. Working on legacy and citizen science data have driven attention to robustness of inference and methodology.
Felix Singleton Thorn
Felix is a PhD student in the School of Psychological Sciences, University of Melbourne, with a background in quantative psychology and reserach methods. Felix's research examines how people plan, report and interpret the results of experiments.
Elise is a PhD student at the School of BioSciences, University of Melbourne, with a background in applied ecology. Elise is investigating the transparency and reproducibility ecological models in conservation decision-making and ecological management.
- reasoning team
Mark Burgman (Principal Investigator)
Mark Burgman is the editor of two books and the author of seven, including Risks and Decisions for Conservation and Environmental Management (Cambridge University Press, 2005) and Trusting judgements: How to get the best out of experts (Cambridge University Press, 2015). In addition, he has published over 250 refereed papers and more than 70 reviewed reports and commentaries. His book on Risks and Decisions outlined the foundations for a range of methods relevant to decision making under uncertainty and foreshadowed the importance of expert judgement and elicitation in empirical studies. In the 1990s, he one was one of the early figures in the development of methods for dealing with the human dimensions of environmental management. From 2006, at the University of Melbourne he led the Australian Centre of Excellence for Risk Analysis and then the Centre of Excellence for Biosecurity Risk Analysis. In 2016, he took up the position of Director of the Centre for Environmental Policy at Imperial College London. He has been the editor-in-chief of the journal Conservation Biology since 2013. The impact factor of his publications (Google Scholar) is 65 and his work has been cited more than 16,000 times.
Martin Bush is a Research Fellow in the School of Historical and Philosophical Studies at the University of Melbourne with expertise in the cultural history of popular science and professional experience in science communication and the museum sector. Particular interests include planetariums, public reasoning practices, the science communication work of the Ngarrindjeri Australian David Unaipon and popular astronomy in Australia in the era of the lantern slide. His recent PhD from Swinburne University is on popular astronomy in Australia in the era of the lantern slide and his essay from the thesis on the Proctor-Parkes affair was a joint winner of the 2016 Mike Smith Student Prize for History of Australian Science.
Eden is a research fellow in Fiona Fidler’s meta-research group at the University of Melbourne. In this project, Eden will focus on investigating the reasoning involved in expert assessments of the replicability, reproducibility, and robustness of scientific claims, as well how concepts such as replicability are used within open-science communities. Eden is also collaborating on a digital-ethnography project exploring the sociotechnical dynamics involved in the open-source development of decentralised technologies by distributed communities. These projects build on Eden’s PhD (2018) research on the historical interdependence of two scientific concepts and their current uses as independent tools in neuroscience experiments.
Fallon is a research fellow in Fiona Fidler's meta-research group at the University of Melbourne. Her expertise is in science communication, qualitative analysis, and history and philosophy of science. Fallon has worked in science communication and qualitative research roles for the Faculty of Science and the Centre for Excellence for Biosecurity Risk Analysis (CEBRA) at the University of Melbourne; and the Royal College of Paediatrics & Child Health in London. Fallon will undertake the research engagement activity for this project, as well as assist in the qualitative analysis of expert reasoning that this project will undertake. Fallon's PhD research was to develop and explore a prosopography of European medical migrants in mid-twentieth century Australia, using their lives to understand the ways in which local/national domains of medical practice develop and are sustained.
- elicitation and aggregation team
Bonnie Wintle (Principal Investigator)
Bonnie is a Research Fellow in the School of Biosciences at the University of Melbourne, and a senior researcher in the Interdisciplinary MetaResearch Group (IMeRG). She develops structured methods for eliciting and aggregating quantitative and qualitative judgements from groups of experts, to support better decision and policy making. She has pioneered empirical research on the best ways to obtain more accurate group estimates of fact, and applied protocols for eliciting quantitative, probabilistic and qualitative judgements from expert groups that have informed real-world decisions (e.g., to underpin surveillance systems used in industry). She has a background in environmental science and ecology, a history of working closely with philosophers, mathematicians and psychologists, and extensive experience managing interdisciplinary expert groups.
Marissa is Decision Science Research Fellow in the School of BioSciences at the University of Melbourne. Her background is applied mathematics and her PhD (2014) examined the theoretical underpinnings of expert judgement, and methods for scoring and aggregation.
Anca is a Senior Research Fellow based at the Centre of Excellence for Biosecurity Risk Analysis (CEBRA) at the University of Melbourne. Her background is in mathematics and risk and environmental modelling. She has a PhD in Applied Probability from the Technical University of Delft (TU Delft). She was instrumental in building a COST European network for structured expert judgement elicitation and aggregation, and related standards for the European Food Safety Authority (EFSA).
Libby is Deputy Director of the Centre for Economic and Environmental Research at the University of Melbourne. Her work focuses on applying participatory approaches to different decision-making contexts. She brings together skills in decision theory, risk assessment, expert elicitation, facilitation, and model development. She is a highly experienced facilitator, and will guide workshop design on the current project.
- research support team
Victoria is a Research Associate at the Centre of Environmental and Economic Research (CEER) at the University of Melbourne, with 10 years’ experience as a consultant and project manager. She is concurrently completing her PhD (to be submitted in Nov 2018) in structured expert judgement and decision making.
Daniel originally trained as a radiation therapist at Epworth hospital in Melbourne, working both clinically and in a research support role between 2012 and 2017. Following his position at Epworth hospital Daniel worked as a research coordinator at Peter MacCallum Cancer Centre managing a large portfolio of national and international radiation oncology clinical trials. He is the lead author on multiple papers investigating novel radiotherapy treatment techniques for prostate and breast cancer, as well as papers examining ethical issues in scientific publishing. Currently he is completing a PhD investigating the quality and integrity of published radiation oncology and medical physics research within A/Prof Fiona Fidler’s Interdisciplinary Meta-Research Group (IMeRG) at the University of Melbourne.
Andrew is a research assistant within IMeRG at the University of Melbourne. He has recently completed a Graduate Diploma of Psychology at Deakin University and is intending to commence a PhD in 2020. Andrew’s research interests include the history and philosophy of science, improving science practices, and improving the quality of public engagement with science.
- tech team
Got questions about our project? You can contact at repliCATSemail@example.com.
However, here’s a list of frequently asked questions that might help.
- What is “replication” and “reproducibility”?
Confusingly, when people talk about "replication" and "reproducibility", they sometimes are talking about slightly different things. For us, a replication, specifically, a direct replication is to independently repeat a study using the procedures or methods of the original study as closely as possible to see if the same findings are obtained. Reproduction, specifically, a computational reproduction is to re-analyse an existing data set to see if the statistical outputs originally reported in a study can be recovered. If a study computationally reproduces, it suggests a minimum standard for assessing the value of a scientific claim, particularly when a direct replication of a study is not feasible.
- Does repliCATS stand for something?
Yes. The “CATS” in repliCATS is an acronym for Collaborative Assessment for Trustworthy Science.
- Who is part of your research team?
We are an interdisciplinary research team based predominantly at the University of Melbourne. You can meet the research team here.
- What are the aims of the repliCATS project?
We are developing and testing methods to elicit accurate predictions about the likely replicability and reproducibility of published research claims in the social sciences. As you may be aware, some large scale, crowdsourced replication projects have alerted us to the possibility that replication success rates may be lower than we once thought. Our project will assist with the development of efficient methods for critically evaluating the evidence base of social science research.
- What is the IDEA protocol?
The IDEA protocol is a structured protocol for eliciting expert judgments based on the Delphi process. IDEA stands for Investigate, Discuss, Estimate, Aggregate. Applying the IDEA protocol involves recruiting a diverse group of experts to answer questions with probabilistic or quantitative responses. Experts first investigate the questions and clarify meanings of terms, reducing variation caused by linguistic ambiguity. They provide their private, individual estimate, using a 3- or 4-step method (highest, lowest, best guess). The group’s private estimates are revealed; group members can then see how their estimates sit in relation to others. The group discusses the results, shares information and cross-examines reasoning and evidence. Group members individually provide a second and final private estimate. These second-round estimates are then combined using mathematical aggregation. The strengths of the IDEA protocol in eliciting predictions of the likely replicability of research claims lies in the stepped, structured nature of the approach. The feedback and discussion components of the IDEA protocol both function to reduce overconfidence in estimates, which is a known limitation of expert elicitation methods. The discussion component of the IDEA protocol also allows experts to account for private information which could substantially alter the likely replicability assessment of a research claim.
- Can I collaborate on this project?
Yes! We hope to crowdsource expert judgements from a diverse range of participants in the following broad disciplines:
- political science
- public administration
- marketing, and
If you are interested in participating, subscribe to our mailing list so we can contact you when we are ready to begin the next phase of the project.
- If I participate, what’s in it for me?
Your participation will help us to refine methods for predicting the replicability of social and behavioural science claims. Any data we collect could drastically change the way we think about published research evidence. For individuals participants, it also provides the opportunity to develop your skills, through peer interactions, and to become more critical consumers of the research literature.
- Where I can get more information about the project?
Keep up-to-date with our project
It is early days for us! We are currently building a custom platform for the repliCATS project, and expect to pilot local workshops shortly.
For now, you can use the form below to subscribe to updates. (Don’t worry, we won’t spam you.)
Form not working?
Privacy Collection Notice – the repliCATS project.
The information on this form is being collected by the repliCATS project, a research group at the University of Melbourne. You can contact us at repliCATSfirstname.lastname@example.org.
The information you provide will be used to communicate with you about the repliCATS project. The information will be used by authorised staff for the purpose for which it was collected, and will be protected against unauthorised access and use.
Melbourne awarded up to US$6.5m to evaluate experts’ IDEAs about reproducibility and replicability
Media release | Wednesday, 3 April 2019 In response to the 'replication crisis' in a number of scientific fields, a major international research program seeks to develop Artificial Intelligence to help evaluate the credibility of scientific evidence we use to make decisions. The University of Melbourne is currently the only Australian team selected by the US Government’s Defense Advanced Research Project Agency …3 April, 2019 Media release
Event: Ethics, Evidence & Scientific Reform
Friday 5 April | 1-3pm | Forum theatre, Arts West Join an interdisciplinary panel of local and international speakers to discuss the role of ethics in reforming scientific practice. Inappropriate statistical practices, poor research design, and other questionable research practices has led to the ‘replication crisis’ in many scientific disciplines. The challenge we face is how to effect change – how to …22 March, 2019 Events, Event
Professor Brian Nosek from the Center for Open Science (COS) to visit Melbourne in April
Brian Nosek is the co-Founder and Executive Director of the Center for Open Science (http://cos.io/) that operates the Open Science Framework (http://osf.io/). COS is enabling open and reproducible research practices worldwide. Brian will be giving a free public lecture on Thursday, 4 April at the University of Melbourne on the gap between scientific values and practice. You can read the abstract and …8 March, 2019 Open science, E...