FAQs

Got questions about our project? You can contact at repliCATS-contact@unimelb.edu.au.

However, here’s a list of frequently asked questions that might help.

About the project & SCORE program

  • What is “replication” as defined for this project?

    Replication, along with many other related terms like reproducibility, are contested. That is, they have multiple meanings.

    For this project, our working definition of a direct replication as a replication that follows the methods of the original study with a high degree of similarity, varying aspects only where there is a high degree of confidence that they are not relevant to the research claim. The aim of a direct replication is to improve confidence in the reliability and validity of an experimental finding by starting to account for things such as sampling error, measurement artefacts, and questionable research practices.

  • Does repliCATS stand for something?

    Yes. The “CATS” in repliCATS is an acronym for Collaborative Assessment for Trustworthy Science.

  • Who is part of your research team?

    We are an interdisciplinary research team based predominantly at the University of Melbourne. You can meet the research team here.

  • What are the aims of the repliCATS project?

    We are developing and testing methods to elicit accurate predictions about the likely replicability of published research claims in the social sciences. As you may be aware, some large scale, crowdsourced replication projects have alerted us to the possibility that replication success rates may be lower than we once thought. Our project will assist with the development of efficient methods for critically evaluating the evidence base of social science research.

  • What is the IDEA protocol?

    The IDEA protocol is a structured protocol for eliciting expert judgments based on the Delphi process. IDEA stands for Investigate, Discuss, Estimate, Aggregate.

    Applying the IDEA protocol involves recruiting a diverse group of experts to answer questions with probabilistic or quantitative responses. Experts first investigate the questions and clarify meanings of terms, reducing variation caused by linguistic ambiguity. They provide their private, individual estimate, using a 3- or 4-step method (highest, lowest, best guess). The group’s private estimates are revealed; group members can then see how their estimates sit in relation to others. The group discusses the results, shares information and cross-examines reasoning and evidence. Group members individually provide a second and final private estimate. These second-round estimates are then combined using mathematical aggregation.

    The strengths of the IDEA protocol in eliciting predictions of the likely replicability of research claims lies in the stepped, structured nature of the approach. The feedback and discussion components of the IDEA protocol both function to reduce overconfidence in estimates, which is a known limitation of expert elicitation methods. The discussion component of the IDEA protocol also allows experts to account for private information which could substantially alter the likely replicability assessment of a research claim.

    This protocol, developed at the University of Melbourne, has been found to improve judgements under uncertainty. IDEA stands for “Investigate”, “Discuss”, “Estimate” and “Aggregate”, the four steps in the process of this elicitation.

    More information on the IDEA protocol can be found ​here​ (external link to: Methods Blog).

  • Can I participate in this project?

    Yes! We hope to crowdsource expert judgements from a diverse range of participants in the following broad disciplines:

    • criminology
    • economics
    • education
    • political science
    • psychology
    • public administration
    • marketing, and
    • sociology.

    If you are interested in participating, express interest using this form and we will get back to you.

  • If I participate, what’s in it for me?

    Your participation will help us to refine methods for predicting the replicability of social and behavioural science claims. Any data we collect could drastically change the way we think about published research evidence. For individuals participants, it also provides the opportunity to develop your skills, through peer interactions, and to become more critical consumers of the research literature.

    Our first workshop was held in July 2019 in Rotterdam, with over 200 participants over two days. Our participants reported that they found the experience valuable and enjoyed thinking about replicability of published research evidence. Additionally, early career researchers said participating in the workshop improved their critical appraisal (or peer review) skills, and they enjoyed comparing their judgements against diverse individuals (from discipline to career stage) in their group.

  • How are the 3,000 research claims chosen?

    The Center for Open Science (USA) are selecting the 3,000 research claims, as a subset of a larger set of 30,000 published papers in the social and behavioural sciences that are in scope for the SCORE program. These are:

    • criminology
    • economics
    • education
    • political science
    • psychology
    • public administration
    • marketing, and
    • sociology.

    These claims will be drawn from the following journals.

    Criminology

     

    Marketing/Organisational Behaviour

    • Criminology
    • Law and Human Behavior

     

     

     

     

     

     

    •  Journal of Consumer Research
    • Journal of Marketing
    • Journal of Marketing Research
    • Journal of Organizational Behavior
    • Journal of the Academy of Marketing Science
    • Organizational Behavior and Human Decision Processes
         

    Economics

     

    Political Science

    • American Economic Journal: Applied Economics
    • American Economic Revie
    • Econometrica 
    • Experimental Economics
    • Journal of Finance
    • Journal of Financial Economics
    • Journal of Labor Economics
    • Quarterly Journal of Economics
    • Review of Financial Studies

     

    • American Political Science Review
    • British Journal of Political Science
    • Comparative Political Studies
    • Journal of Conflict Resolution
    • Journal of Experimental Political Science
    • Journal of Political Economy
    • World Development
    • World Politics

     

         

    Education

     

    Psychology

    • American Educational Research Journal
    • Computers and Education
    • Contemporary Educational Psychology
    • Educational Researcher
    • Exceptional Children
    • Journal of Educational Psychology
    • Learning and Instruction

     

     

     

     

     

     

    • Child Development
    • Clinical Psychological Science
    • Cognition
    • European Journal of Personality
    • Evolution and Human Behavior
    • Journal of Applied Psychology
    • Journal of Consulting and Clinical Psychology
    • Journal of Environmental Psychology
    • Journal of Experimental Psychology: General
    • Journal of Experimental Social Psychology
    • Journal of Personality and Social Psychology
    • Psychological Science
         

    Health related

     

    Public Administration

    • Health Psychology
    • Psychological Medicine
    • Social Science and Medicine

     

    • Journal of Public Administration Research and Theory
    • Public Administration Review
         

    Management

     

    Sociology

    • Academy of Management Journal
    • Journal of Business Research
    • Journal of Management
    • Leadership Quarterly
    • Management Science
    • Organization Science

     

    • American Journal of Sociology
    • American Sociological Review
    • Demography
    • European Sociological Review
    • Journal of Marriage and Family
    • Social Forces

  • From which journals are the 3,000 research claims chosen?

    The Center for Open Science (USA) are selecting the 3,000 research claims from the following journals. 

    Criminology

     

    Marketing/Organisational Behaviour

    • Criminology
    • Law and Human Behavior

     

     

     

     

     

     

    •  Journal of Consumer Research
    • Journal of Marketing
    • Journal of Marketing Research
    • Journal of Organizational Behavior
    • Journal of the Academy of Marketing Science
    • Organizational Behavior and Human Decision Processes
         

    Economics

     

    Political Science

    • American Economic Journal: Applied Economics
    • American Economic Revie
    • Econometrica 
    • Experimental Economics
    • Journal of Finance
    • Journal of Financial Economics
    • Journal of Labor Economics
    • Quarterly Journal of Economics
    • Review of Financial Studies

     

    • American Political Science Review
    • British Journal of Political Science
    • Comparative Political Studies
    • Journal of Conflict Resolution
    • Journal of Experimental Political Science
    • Journal of Political Economy
    • World Development
    • World Politics

     

         

    Education

     

    Psychology

    • American Educational Research Journal
    • Computers and Education
    • Contemporary Educational Psychology
    • Educational Researcher
    • Exceptional Children
    • Journal of Educational Psychology
    • Learning and Instruction

     

     

     

     

     

     

    • Child Development
    • Clinical Psychological Science
    • Cognition
    • European Journal of Personality
    • Evolution and Human Behavior
    • Journal of Applied Psychology
    • Journal of Consulting and Clinical Psychology
    • Journal of Environmental Psychology
    • Journal of Experimental Psychology: General
    • Journal of Experimental Social Psychology
    • Journal of Personality and Social Psychology
    • Psychological Science
         

    Health related

     

    Public Administration

    • Health Psychology
    • Psychological Medicine
    • Social Science and Medicine

     

    • Journal of Public Administration Research and Theory
    • Public Administration Review
         

    Management

     

    Sociology

    • Academy of Management Journal
    • Journal of Business Research
    • Journal of Management
    • Leadership Quarterly
    • Management Science
    • Organization Science

     

    • American Journal of Sociology
    • American Sociological Review
    • Demography
    • European Sociological Review
    • Journal of Marriage and Family
    • Social Forces

  • How can I get more information about this project?

    You can express interest in assessing claims, or subscribe to our mailing list. 

    You can also follow us on twitter, @replicats.

    Or, you can send us an e-mail at repliCATS-contact@unimelb.edu.au.

Answering claims

  • I want to assess claims. What do I need to do?

    Great! You can create an account and log on to our platform by visiting: https://score.eresearch.unimelb.edu.au 

    The first step when you create an account will be a short survey which includes a plain language statement, obtaining your consent, and some demographic information about you.

    You might also find the following pages useful to:

  • I don't understand a term on the platform. Is there a glossary?

    Yes there is, click here repliCATS glossary.

    If you think there's a term missing or defined incorrectly, send us an e-mail to: repliCATS-contact@unimelb.edu.au

  • I'm on the platform, how do I assess a claim?

    There is a load of information we've prepared to help you navigate the website, as well as get comfortable about answering claims.

    Check out the resources page for videos, handy guides, and a whole bunch of additional information.