Got questions about our project? You can contact at repliCATSemail@example.com.
However, here’s a list of frequently asked questions that might help.
- What is “replication” and “reproducibility”?
Confusingly, when people talk about "replication" and "reproducibility", they sometimes are talking about slightly different things. For us, a replication, specifically, a direct replication is to independently repeat a study using the procedures or methods of the original study as closely as possible to see if the same findings are obtained. Reproduction, specifically, a computational reproduction is to re-analyse an existing data set to see if the statistical outputs originally reported in a study can be recovered. If a study computationally reproduces, it suggests a minimum standard for assessing the value of a scientific claim, particularly when a direct replication of a study is not feasible.
- Does repliCATS stand for something?
Yes. The “CATS” in repliCATS is an acronym for Collaborative Assessment for Trustworthy Science.
- Who is part of your research team?
We are an interdisciplinary research team based predominantly at the University of Melbourne. You can meet the research team here.
- What are the aims of the repliCATS project?
We are developing and testing methods to elicit accurate predictions about the likely replicability and reproducibility of published research claims in the social sciences. As you may be aware, some large scale, crowdsourced replication projects have alerted us to the possibility that replication success rates may be lower than we once thought. Our project will assist with the development of efficient methods for critically evaluating the evidence base of social science research.
- What is the IDEA protocol?
The IDEA protocol is a structured protocol for eliciting expert judgments based on the Delphi process. IDEA stands for Investigate, Discuss, Estimate, Aggregate. Applying the IDEA protocol involves recruiting a diverse group of experts to answer questions with probabilistic or quantitative responses. Experts first investigate the questions and clarify meanings of terms, reducing variation caused by linguistic ambiguity. They provide their private, individual estimate, using a 3- or 4-step method (highest, lowest, best guess). The group’s private estimates are revealed; group members can then see how their estimates sit in relation to others. The group discusses the results, shares information and cross-examines reasoning and evidence. Group members individually provide a second and final private estimate. These second-round estimates are then combined using mathematical aggregation. The strengths of the IDEA protocol in eliciting predictions of the likely replicability of research claims lies in the stepped, structured nature of the approach. The feedback and discussion components of the IDEA protocol both function to reduce overconfidence in estimates, which is a known limitation of expert elicitation methods. The discussion component of the IDEA protocol also allows experts to account for private information which could substantially alter the likely replicability assessment of a research claim.
- Can I collaborate on this project?
Yes! We hope to crowdsource expert judgements from a diverse range of participants in the following broad disciplines:
- political science
- public administration
- marketing, and
If you are interested in participating, subscribe to our mailing list so we can contact you when we are ready to begin the next phase of the project.
- If I participate, what’s in it for me?
Your participation will help us to refine methods for predicting the replicability of social and behavioural science claims. Any data we collect could drastically change the way we think about published research evidence. For individuals participants, it also provides the opportunity to develop your skills, through peer interactions, and to become more critical consumers of the research literature.
- Where I can get more information about the project?