Home page

The repliCATS project aim is to crowdsource predictions about the replicability of published research. In August 2020, we completed assessing the first 3000 claims from eight social & behavioural science fields. In 2021, we will begin phase 2 of the SCORE program.

Why is it important to predict replicability? In the past 10 years, science has faced a ‘replication crisis’. Researchers have been unable to replicate the results of several landmark studies in medicine, psychology, economics and other fields, causing many to question the scientific evidence base we use to make decisions. But, we can’t afford to test and replicate every piece of research before it’s published – it will be too costly and time consuming.

“If we can accurately predict credible research, our project could transform how end-users – from academics to policy makers – can assess the reliability of social scientific research.”

–– Prof Fiona Fidler, chief investigator of repliCATS project

The repliCATS project is part of a research program called SCORE, funded by DARPA, that eventually aims to build automated tools that can rapidly and reliably assign confidence scores to social science research claims.

Our approach

The “CATS” in repliCATS stands for Collaborative Assessment for Trustworthy Science.

The repliCATS project uses a structured iterative approach for evaluating the replicability of research claims, called the IDEA protocol, deployed on a custom-built technical platform.

For each of 3000 claims, four or more group members first Investigate the claim and provide an initial private estimate of the probability of successful replication, together with qualitative reasons behind their judgments. Group members then Discuss, provide a second, private Estimate in light of discussion, and the repliCATS team Aggregates individual judgements using a diverse portfolio of mathematical methods, some incorporating characteristics of reasoning, engagement and uncertainty.

As of August 2020, over 500 participants have participated in our project through face-to-face workshops and virtually using our remote platform. We have collected group estimates of the replicability of 3,000 published social scientific research claims in: Business research, Criminology, Economics, Education, Political Science, Psychology, Public Administration, and Sociology.

the repliCATS project news feed

The repliCATS project is led by Prof Fiona Fidler. We are a group of interdisciplinary researchers from the School of BioSciences, School of Historical and Philosophical Studies, and the Melbourne School of Engineering at the University of Melbourne, with collaboration from the Centre for Environmental Policy at Imperial College London.

the repliCATS project aim is to crowdsource predictions about the reliability of published research.

why is it important to predict replicability?

In the past 10 years, science has faced a ‘replication crisis’. Researchers have been unable to replicate the results of several landmark studies in medicine, psychology, economics and other fields, causing many to question the scientific evidence base we use to make decisions.

But, we can’t afford to test and replicate every piece of research before it’s published – it will be too costly and time consuming.

If we can accurately predict credible research, our project could transform how end-users – from academics to policy makers – can assess the reliability of social scientific research.

–– Prof Fiona Fidler, chief investigator of repliCATS project

about our project

The “CATS” in repliCATS stands for Collaborative Assessment for Trustworthy Science.

We aim to estimate the replicability of 3,000 published social scientific research claims in: Business research, Criminology, Economics, Education, Political Science, Psychology, Public Administration, and Sociology.

We hope to develop more accurate, better-calibrated techniques to elicit estimates of the replicability of social & behavioural science evidence.

Our approach is to adapt and deploy the IDEA protocol developed here at the University of Melbourne to elicit group judgements for the likely replicability of 3,000 research claims.

The research we will undertake as part of the repliCATS project will include the largest ever empirical study on how scientists reason about other scientists’ work, and what factors makes them trust it.

We have built a custom online platform to deploy the IDEA protocol. This platform will have a life beyond the repliCATS project: it will be able to be used in the future to enhance group judgements on a wide range of topics, in a number of disciplines.

We’re an interdisciplinary team, based predominantly at the University of Melbourne, but we have people from Germany, Netherlands, UK and the USA working on our project too.

The repliCATS project is a part of Prof Fiona Fidler’s Interdisciplinary MetaResearch Group (IMeRG). You can check out our team bios below.

Team bios

leadership team
Fiona Fidler

Fiona is a professor at the University of Melbourne, with a joint appointment in the Schools of BioSciences and History and Philosophy of Science. She is broadly interested in how experts, including scientists, make decisions and change their minds. Her past research has examined how methodological change occurs in different disciplines, including psychology, medicine and ecology, and developed methods for eliciting reliable expert judgements to improve decision making. She originally trained as a psychologist, and maintains a strong interest in psychological methods. She also has an abiding interest is statistical controversies, for example, the ongoing debate over Null Hypothesis Significance Testing. She is a current Australian Research Council Future Fellow, and leads the University of Melbourne’s Interdisciplinary MetaResearch Group (IMeRG), and the lead PI of the repliCATS project.

Hannah Fraser

Hannah is the research program coordinator for the repliCATS project. She is a research fellow at the University of Melbourne working in Fiona Fidler’s meta-research lab, IMeRG. She is lead author of Questionable Research Practices in Ecology and Evolution (Fraser et al. 2018), which has received widespread attention (preprint downloaded 679 times). During her PhD, Hannah also gained expert elicitation experience.

Ross Pearson

Ross Pearson is a digital supply chain transformation leader that has been the technical lead for telecommunications and mining transformations. As a delivery specialist, Ross ensures that large projects and transformation implementations are realised. In addition to his supply chain experience, Ross has worked on numerous University research projects and is currently working with Melbourne University to ensure the success on the SCORE project. Ross has recently completed an honours in Computer Science with a focus on Artificial Intelligence.
reproducibility team
Peter Vesk (Principal Investigator)

Peter is an Associate Professor and Reader in the School of BioSciences at University of Melbourne. He has a long history of working on generalization and reliability of scientific knowledge before it was known as reproducibility, starting in plant ecology. He is an Associate Editor at Journal of Ecology, the most highly ranked journal in plant ecology. As a founding editor of Hot Topics in Ecology, designed to provide evidence based statements on ecological topics relevant to policy and management, he is keenly interested in participatory methods of providing reliable scientific knowledge. He has > 100 journal articles, with >4700 citations and H=36 (Scopus). Vesk’s research focus is gathering, formalizing and generalizing knowledge for management. This entails attention to methodology of data collection, use and model evaluation. Working on legacy and citizen science data have driven attention to robustness of inference and methodology.

Elliot Gould

Elliot is a PhD student at the School of BioSciences, University of Melbourne, with a background in applied ecology. Elliot is investigating the transparency and reproducibility ecological models in conservation decision-making and ecological management.

Aaron Wilcox

Aaron has spent the last three years shifting gears from a technical production industry, into psychological science degree. Transferring skills of project management into areas of research in data and neuroscience. Recently returning from studying abroad at Maastricht University in the Netherlands where, he also attend the Replicats and SIPS conference in Rotterdam. Aaron also works as a research assistant at Deakin University under Emma Sciberras. Aaron is an R acolyte and an open science advocate and will be continuing studies into fourth year and beyond.

Charles T. Gray

Charles spent twenty years as a pianist, and is now retraining in mathematical science. Her interdisciplinary computational metascience doctoral work-in-progress is a rumination on open reproducible computing in mathematical science towards achieving a good enough level of confidence in the implementation of statistical algorithms. Charles, having completed degrees in literature, music, and mathematics, sees mathematics as a liberal art that wanders into philosophy.

David Wilkinson

David is a PhD student in the School of BioSciences at the University of Melbourne. David has a background in quantitative ecology. His PhD focuses on the computational, inferential, and predictive performance of joint species distribution models.What my role is for the research assistance work is to be determined so I haven't added anything on that. Somewhere between working with Elise and Aaron on the reproducibility team and being a general quantitative helper across various parts of the group.
reasoning team
Mark Burgman (Principal Investigator)

Mark Burgman is the editor of two books and the author of seven, including Risks and Decisions for Conservation and Environmental Management (Cambridge University Press, 2005) and Trusting judgements: How to get the best out of experts (Cambridge University Press, 2015). In addition, he has published over 250 refereed papers and more than 70 reviewed reports and commentaries. His book on Risks and Decisions outlined the foundations for a range of methods relevant to decision making under uncertainty and foreshadowed the importance of expert judgement and elicitation in empirical studies. In the 1990s, he one was one of the early figures in the development of methods for dealing with the human dimensions of environmental management. From 2006, at the University of Melbourne he led the Australian Centre of Excellence for Risk Analysis and then the Centre of Excellence for Biosecurity Risk Analysis. In 2016, he took up the position of Director of the Centre for Environmental Policy at Imperial College London. He has been the editor-in-chief of the journal Conservation Biology since 2013. The impact factor of his publications (Google Scholar) is 65 and his work has been cited more than 16,000 times.

Martin Bush

Martin Bush is a Research Fellow in the School of Historical and Philosophical Studies at the University of Melbourne with expertise in the cultural history of popular science and professional experience in science communication and the museum sector. Particular interests include planetariums, public reasoning practices, the science communication work of the Ngarrindjeri Australian David Unaipon and popular astronomy in Australia in the era of the lantern slide. His recent PhD from Swinburne University is on popular astronomy in Australia in the era of the lantern slide and his essay from the thesis on the Proctor-Parkes affair was a joint winner of the 2016 Mike Smith Student Prize for History of Australian Science.

Eden Smith

Eden is a research fellow in Fiona Fidler’s meta-research group at the University of Melbourne. In this project, Eden will focus on investigating the reasoning involved in expert assessments of the replicability, reproducibility, and robustness of scientific claims, as well how concepts such as replicability are used within open-science communities. Eden is also collaborating on a digital-ethnography project exploring the sociotechnical dynamics involved in the open-source development of decentralised technologies by distributed communities. These projects build on Eden’s PhD (2018) research on the historical interdependence of two scientific concepts and their current uses as independent tools in neuroscience experiments.

Fallon Mody

Fallon is a research fellow in Fiona Fidler's meta-research group at the University of Melbourne. Her expertise is in science communication, qualitative analysis, and history and philosophy of science. Fallon has worked in science communication and qualitative research roles for the Faculty of Science and the Centre for Excellence for Biosecurity Risk Analysis (CEBRA) at the University of Melbourne; and the Royal College of Paediatrics & Child Health in London. Fallon will undertake the research engagement activity for this project, as well as assist in the qualitative analysis of expert reasoning that this project will undertake. Fallon's PhD research was to develop and explore a prosopography of European medical migrants in mid-twentieth century Australia, using their lives to understand the ways in which local/national domains of medical practice develop and are sustained.
elicitation and aggregation team
Bonnie Wintle (Principal Investigator)

Bonnie is a Research Fellow in the School of Biosciences at the University of Melbourne, and a senior researcher in the Interdisciplinary MetaResearch Group (IMeRG). She develops structured methods for eliciting and aggregating quantitative and qualitative judgements from groups of experts, to support better decision and policy making. She has pioneered empirical research on the best ways to obtain more accurate group estimates of fact, and applied protocols for eliciting quantitative, probabilistic and qualitative judgements from expert groups that have informed real-world decisions (e.g., to underpin surveillance systems used in industry). She has a background in environmental science and ecology, a history of working closely with philosophers, mathematicians and psychologists, and extensive experience managing interdisciplinary expert groups.

Marissa McBride

Marissa is Decision Science Research Fellow in the School of BioSciences at the University of Melbourne. Her background is applied mathematics and her PhD (2014) examined the theoretical underpinnings of expert judgement, and methods for scoring and aggregation.

Anca Hanea

Anca is a Senior Research Fellow based at the Centre of Excellence for Biosecurity Risk Analysis (CEBRA) at the University of Melbourne. Her background is in mathematics and risk and environmental modelling. She has a PhD in Applied Probability from the Technical University of Delft (TU Delft). She was instrumental in building a COST European network for structured expert judgement elicitation and aggregation, and related standards for the European Food Safety Authority (EFSA).

Libby Rumpff

Libby is Deputy Director of the Centre for Economic and Environmental Research at the University of Melbourne. Her work focuses on applying participatory approaches to different decision-making contexts. She brings together skills in decision theory, risk assessment, expert elicitation, facilitation, and model development. She is a highly experienced facilitator, and will guide workshop design on the current project.
research support team
Raquel Ashton

Raquel is a qualified wildlife veterinarian, expert elicitor and shadow editor for the journal, Biological Conservation. She is currently monitoring the health of repliCATS as the IDEA workflow co-ordinator.

Raquel has recently joined the team, so a photo is on its way.

Cassie Watts

Cassie has ten years experience as a business manager at the University of Melbourne and joins the repliCATS project team as Finance Manager with a wealth of experience managing small and large research grants and consultancies.

Daniel Hamilton

Daniel originally trained as a radiation therapist at Epworth hospital in Melbourne, working both clinically and in a research support role between 2012 and 2017. Following his position at Epworth hospital Daniel worked as a research coordinator at Peter MacCallum Cancer Centre managing a large portfolio of national and international radiation oncology clinical trials. He is the lead author on multiple papers investigating novel radiotherapy treatment techniques for prostate and breast cancer, as well as papers examining ethical issues in scientific publishing. Currently he is completing a PhD investigating the quality and integrity of published radiation oncology and medical physics research within A/Prof Fiona Fidler’s Interdisciplinary Meta-Research Group (IMeRG) at the University of Melbourne.

Andrew Head

Andrew is a research assistant within IMeRG at the University of Melbourne. He has recently completed a Graduate Diploma of Psychology at Deakin University and is intending to commence a PhD in 2020. Andrew’s research interests include the history and philosophy of science, improving science practices, and improving the quality of public engagement with science.

Mel Ross

Mel originally trained as a physiotherapist and has had a varied career working in hospital rehabilitation for over 17 years. More recently she has moved away from health care and has been working in Business Development and Sales. She has joined the team to assist with Admin Support.

Nicholas (Nick) Dempsey

Nick is a graphics designer, and he has designed all the badges participants are awarded on our research platform. Nick graduated from a Digital Media Design degree awarded at Swinburne University in 2019. Nick's interests in design revolve around visual communication, typography, motion graphics, video and working with brands. In his spare time, he is an avid collector of vinyl records and loves photography and technology.
tech team
The tech team, led by Professor Richard Sinnott, are part of the University of Melbourne's eResearch Group: https://www.eresearch.unimelb.edu.au
external consultants
Mathew Goodwin

Matthew is a founding and key faculty member of a new doctoral program in Personal Health Informatics (PHI) and Director of the Computational Behavioral Science Laboratory (CBSL) at Northeastern University. He is also a visiting associate professor in the Department of Biomedical Informatics at Harvard Medical School (2018-2020), the former director of Clinical Research at the MIT Media Lab (2008-2011), and adjunct associate research scientist in the Department of Psychiatry & Human Behavior at Brown University. Mathew has 20 years of research and clinical experience working with children and adults on the autism spectrum and developing and evaluating innovative technologies for behavioral assessment and intervention, including naturalistic video and audio capture, telemetric physiological monitors, wireless accelerometry sensors, and digital video/facial recognition systems.

James Heathers

James is a research scientist working with Professor Matthew Goodwin on bio-signals and meta-science research. James’ research is focused on data science and personal health informatics. Prior to joining Northeastern, Heathers worked as a Post-Doctoral Researcher at Poznan University of Medical Sciences researching Electrocardiology. He earned his PhD and MS degrees from the University of Sydney.

Aidan Lyon

Aidan is CEO and co-founder of DelphiCloud and Research Associate in the Institute for Logic, Language and Computation at the University of Amsterdam. He completed his PhD at the Australian National University on the philosophical foundations of probability and has degrees in mathematics and philosophy from the University of Queensland. He has held academic positions at the University of Maryland, the Munich Center for Mathematical Philosophy, the Tilburg Center for Ethics and Philosophy of Science, the University of Vienna, the University of Melbourne, the University of Sydney, and the Australian National University. In addition to being an academic, he has operated as a risk management consultant for the Australian Government and other clients since 2011. Aidan's research is primarily on the philosophical foundations of uncertainty, philosophical psychology, and social epistemology — with a particular focus on the so-called wisdom of crowds.

Alex Marcoci

Alex is a Teaching Assistant Professor of Philosophy at the University of North Carolina at Chapel Hill, a core faculty member in the UNC-Duke Philosophy, Politics and Economics Program and a Visiting Researcher in the Centre for Environmental Policy at Imperial College London. He works at the intersection of formal and applied issues in rationality, decision theory and public policy. For Alex's full bio, visit: http://personal.lse.ac.uk/marcoci/

Shinichi Nakagawa

Shinichi is a professor at UNSW, Sydney (Australia), the Deputy Director of Research at the Evolution & Ecology Research Centre, EERC at UNSW, and an elected member of the Society of Research Synthesis Methods; he is also a Cochrane-review author. In addition to empirical papers on various topics in ecology and evolution, he has published over 60 meta-analyses and 30 methodological papers, which include a new effect size static (lnCVR) and various new methods (e.g., phylogenetic multilevel meta-analysis and R-squared for mixed models). With collobrators such as Tim and Fiona, he has recently established EcoEvoRxiv (a preprint server) for the field of ecology, evolution and coservation biology.

Tim Parker

Tim is an associate professor of Biology at Whitman College in Washington State, USA. He is a leading proponent of transparency as a tool for improving reliability of inferences in ecology and evolutionary biology. Tim also sits on multiple committees at COS (Center for Open Science). He co-organized the Improving Inference in Ecology and Evolution workshop for editors of major journals in these disciplines, and he led the formulation of TTEE (Tools for Transparency in Ecology and Evolution). Tim's interests in improving the reliability of ecological and evolutionary inferences emerged from his work in meta-analysis and his resulting insights into the ways that researchers bias their publication of empirical results. Besides meta-analyses, he has published multiple synthesis, review, and opinion papers related to improving the reliability of inference derived from published literature, and he is collaborating with Fiona Fidler’s group in empirical studies to understand the scope and causes of bias in the ecology and evolutionary biology literature.
repliCATS alum
Victoria Hemming

Victoria was our first local workshop coordinator, and was instrumental in running our first workshop in Rotterdam where we assessed 575 claims. Victoria is currently completing a postdoc in Canada. While working on her PHD, she was a Research Associate at the Centre of Environmental and Economic Research (CEER) at the University of Melbourne, with 10 years’ experience as a consultant and project manager. She finished her PhD in structured expert judgement and decision making.

Felix Singleton Thorn

Felix is a PhD student in the School of Psychological Sciences, University of Melbourne, with a background in quantative psychology and reserach methods. Felix's research examines how people plan, report and interpret the results of experiments.

Participating in our project means making judgements about a published research claim.

What does that mean exactly?

Our project aim is to assess 3000 published research articles and 100 COVID-19 pre-prints in the following disciplines: criminology, economics, education, marketing, management, psychology, public administration, and sociology. The full list of journals the claims are drawn from is listed here.

What we ask you to do is to evaluate the replicability of a claim, that is if you were to follow the methods of the original study with a high degree of similarity do you think we’d find similar results to the original study?

Don’t worry, we don’t ask you to do this alone. Our method involves structured group discussion – each claim is assessed by 3-5 other people, and you get to see what others in your group say before submitting your final judgement.

Why get involved?

Be a part the largest effort to evaluate reliability in the social & behavioural sciences! You’ll also get to:

improve your peer-review & error detection skills
calibrate your judgements & reasoning against your peers

Who can assess claims? Every participant counts – don’t worry about being an expert, we need diverse views

Our method – the IDEA protocol – harnesses the power of structured group discussion in predicting likely replicability. We have built a custom web-based platform to do this. What we ask you to do is to evaluate the replicability of a claim, that is if you were to follow the methods of the original study with a high degree of similarity do you think we’d find similar results to the original study?

Part of the scope of the repliCATS project, and indeed the wider SCORE program is to examine the markers of expertise (e.g. education, experience, domain knowledge), and the role they may play in making good judgements about the likelihood a research claim will replicate.

This means an eligible research participant for our project is someone who has completed or is completing a relevant undergraduate degree, and is over 18 years of age. And, importantly, is interested in making judgements about replicability.

If you would like more information, you can:

—

How to get started on our remote platform

You can assess claims anywhere, anytime!

To create an account, or to login visit: https://score.eresearch.unimelb.edu.au

We have a custom-built platform participants use to assess claims. Every month, we release a mix of new claims across the disciplines that are in scope for the SCORE program. (The first time you create an account, you will be asked to complete a consent form, as well as a short demographics survey and a quiz. All up this takes about 30 minutes.)

As of August 2020, we have completed assessment of the first 3000 claims.

Read more about: repliCATS monthly prizes and rules
View: Monthly winners! (Of repliCATS remote rounds 1-7 which ran from Jan – Jul 2020)

Express interest in assessing COVID-19 claims

The repliCATS project — and the wider SCORE program — has been expanded by DARPA to include 100 COVID-19 research claims from the social and behavioural sciences. Click here to read more and express interest.

Just want to stay up-to-date on the project? Subscribe to our newsletter

We have a quarterly newsletter we send out about our project. By subscribing you’ll get a short, snappy newsletter letting you know what we’ve been up to, and what’s happening with the repliCATS project.

You just sign-up using this form (we won’t spam you)

Privacy Collection Notice – the repliCATS project.

Human ethics application ID: 1853445.1

The information on this form is being collected by the repliCATS project, a research group at the University of Melbourne. You can contact us at repliCATS-contact@unimelb.edu.au.

The information you provide will be used to communicate with you about the repliCATS project. The information will be used by authorised staff for the purpose for which it was collected, and will be protected against unauthorised access and use.

You may access any personal information you have provided to the University by contacting us at repliCATS-contact@unimelb.edu.au. The University of Melbourne is committed to protecting personal information provided by you in accordance with the Privacy and Data Protection Act 2014 (Vic). All information collected by the University is governed by the University’s Privacy Policy. For further information about how the University deals with personal information, please refer to the University’s Privacy Policy or contact the University’s Privacy Officer at privacy-officer@unimelb.edu.auhttps://twitter.com/bg_farrar/status/1147551209173262336?s=20

One of the most frequently asked questions is, “Do I need to be an expert in any individual field to participate in the repliCATS project?” The answer is: No! To participate, you need to be 18 years or older, have completed or be completing an undergraduate degree, and be interested in evaluating research claims in scope for our project.

Below are blocks of frequently asked questions which may help you understand our project and evaluating claims better:

What claims are being assessed? How are they chosen?
Help answering claims
Troubleshooting the platform
$1000 monthly prizes
General questions about the repliCATS project & SCORE

If you still can’t find the answer to your question, you can contact us at repliCATS-contact@unimelb.edu.au.

About SCORE claims

What is a research claim or claim?
In this project we use the word "research claim" or "claim" in a very specific way.

A research claim is a single major finding from a published study (for example, a journal article), as well as details of the methods and results that support this finding. A research claim is not equivalent to an entire article. Sometimes the claim as described in the abstract does not exactly match the claim that is tested. In this case, you should consider the research claim to be that which is described in the inferential test, as the next stage of SCORE will focus on testing the replicability of the test results only.

How are the 3000 claims chosen?

The Center for Open Science (USA) are selecting the 3,000 research claims, as a subset of a larger set of 30,000 published papers in the social and behavioural sciences that are in scope for the SCORE program. These are:

criminology
economics
education
political science
psychology
public administration
marketing, and
sociology.

These claims will be drawn from the following journals.

Criminology		Marketing/Organisational Behaviour
Criminology Law and Human Behavior		Journal of Consumer Research Journal of Marketing Journal of Marketing Research Journal of Organizational Behavior Journal of the Academy of Marketing Science Organizational Behavior and Human Decision Processes

Economics		Political Science
American Economic Journal: Applied Economics American Economic Revie Econometrica Experimental Economics Journal of Finance Journal of Financial Economics Journal of Labor Economics Quarterly Journal of Economics Review of Financial Studies		American Political Science Review British Journal of Political Science Comparative Political Studies Journal of Conflict Resolution Journal of Experimental Political Science Journal of Political Economy World Development World Politics

Education		Psychology
American Educational Research Journal Computers and Education Contemporary Educational Psychology Educational Researcher Exceptional Children Journal of Educational Psychology Learning and Instruction		Child Development Clinical Psychological Science Cognition European Journal of Personality Evolution and Human Behavior Journal of Applied Psychology Journal of Consulting and Clinical Psychology Journal of Environmental Psychology Journal of Experimental Psychology: General Journal of Experimental Social Psychology Journal of Personality and Social Psychology Psychological Science

Health related		Public Administration
Health Psychology Psychological Medicine Social Science and Medicine		Journal of Public Administration Research and Theory Public Administration Review

Management		Sociology
Academy of Management Journal Journal of Business Research Journal of Management Leadership Quarterly Management Science Organization Science		American Journal of Sociology American Sociological Review Demography European Sociological Review Journal of Marriage and Family Social Forces

From which journals are the 3000 claims being chosen?

The Center for Open Science (USA) are selecting the 3,000 research claims from the following journals.

Criminology		Marketing/Organisational Behaviour
Criminology Law and Human Behavior		Journal of Consumer Research Journal of Marketing Journal of Marketing Research Journal of Organizational Behavior Journal of the Academy of Marketing Science Organizational Behavior and Human Decision Processes

Economics		Political Science
American Economic Journal: Applied Economics American Economic Revie Econometrica Experimental Economics Journal of Finance Journal of Financial Economics Journal of Labor Economics Quarterly Journal of Economics Review of Financial Studies		American Political Science Review British Journal of Political Science Comparative Political Studies Journal of Conflict Resolution Journal of Experimental Political Science Journal of Political Economy World Development World Politics

Education		Psychology
American Educational Research Journal Computers and Education Contemporary Educational Psychology Educational Researcher Exceptional Children Journal of Educational Psychology Learning and Instruction		Child Development Clinical Psychological Science Cognition European Journal of Personality Evolution and Human Behavior Journal of Applied Psychology Journal of Consulting and Clinical Psychology Journal of Environmental Psychology Journal of Experimental Psychology: General Journal of Experimental Social Psychology Journal of Personality and Social Psychology Psychological Science

Health related		Public Administration
Health Psychology Psychological Medicine Social Science and Medicine		Journal of Public Administration Research and Theory Public Administration Review

Management		Sociology
Academy of Management Journal Journal of Business Research Journal of Management Leadership Quarterly Management Science Organization Science		American Journal of Sociology American Sociological Review Demography European Sociological Review Journal of Marriage and Family Social Forces

Answering claims – help please!

I want to assess claims. What do I need to do?
Great! You can create an account and log on to our platform by visiting: https://score.eresearch.unimelb.edu.au

The first step when you create an account will be a short survey which includes a plain language statement, obtaining your consent, and some demographic information about you.

You might also find the following pages useful to:
- Find out more about getting involved
- Resources and guides to answering claims
I'm on the platform, how do I assess a claim?
There is a load of information we've prepared to help you navigate the website, as well as get comfortable about answering claims.

Check out the resources page for videos, handy guides, and a whole bunch of additional information.
How long should I spend evaluating a claim?
You can spend as much time as you want, however, we suggest that all up you shouldn't spend more than 30 minutes per claim, for both round one and round two.

During workshops, we use a model of 10 minutes for round one, 15 minutes for discussion and 5 minutes to update and submit your round two.

For virtual groups, the discussion is via commenting and up or down votes. So, if you are working solo or completely virtually (i.e. no real time discussion), we suggest that you spend around 10-15 minutes to complete round one, which includes perusing the paper (there is a link in the lefthand side panel), and spending a bit of extra time to writing down your reasoning. This will help you and the other participants for round two.
There seems to be multiple claims in the paper. Which one do I evaluate?
Sometimes the claim text (in bold) indicates a claim different from that reported in the inferential test results. In this case, all your answers should relate to the inferential test results.

Also, some papers have very few claims and deploy very few tests, others have dozens or hundreds – evaluating ‘all the claims made’ would be incredibly unwieldy. Remember: you only need to evaluate the central claim listed in the claim panel on the right sidebar.
Do you have guides or any resources I can access to help me answer claims?
Yes! Check out our resources page.
Do I need to read the whole paper that is linked to the claim?
Only if you feel like it. We think it is a good idea to look at the paper, and read as much of the paper as is sufficient to help you evaluate the replicability of the central claim presented in the platform.
How do I approach answering the replicability question?
For each claim you evaluate we ask you to estimate the probability that direct replications of this study would find a statistically significant effect in the same direction as the original claim (0-100%). 0 means that you think that a direct replication would never succeed, even by chance. 100 means that you think that a direct replication would never fail, even by chance.

To answer this question, imagine 100 replications of the original study, combined to produce a single, overall replication estimate (e.g., a meta-analysis with no publication bias). How likely is it that the overall estimate will be similar to the original? Note that all replication studies are ‘direct’ replications, i.e., they constitute reasonable tests of the original claim, despite minor changes that may have occurred in methods or procedure. And all replication studies have high power (90% power to detect an effect 50-75% of the original effect size with alpha=0.05, two-sided).

In the text box, we also ask you to note what factors influenced your judgement about whether the claim would successfully replicate, or not. For each of the following, list some factors (dot points are fine):

For your lower bound, think of factors that make successful replications unlikely

For your upper bound, think of factors that make successful replications likely.

For your best estimate, consider the balance of factors.

How will this research claim be replicated?

We cannot answer this question precisely. The selection and replication of claims for the SCORE program is being overseen by the Centre for Open Science, independently of the repliCATS project. See here for more details about this part of the SCORE program.

For SCORE, the intent of a direct replication is to follow the methods of the original study with a high degree of similarity, varying aspects only where there is a high degree of confidence that they are not relevant to the research claim being investigated. However, it is generally impossible to follow a study precisely, and the question as to which aspects matter is a judgement call.

Our best advice is to imagine what kinds of decisions you would face if you were asked to replicate this research claim, and then to consider the effects of making different choices for these decisions. This is one reason why we ask you to consider a set of 100 replications when making your assessment – even though they are 100 direct replications, each might be slightly different. You should consider the effect of these slight variations when making your estimate.

In some instances, a replication may not be able to collect new data, for example, if the claim relates to a specific historical event, like an election. In this case you should consider the different choices that could be made in analysing the data. Again, you should consider the effect of slight variations in these choices when making your estimate of replicability.
I don't understand a term on the platform. Is there a glossary?
Yes there is, click here repliCATS glossary.

If you think there's a term missing or defined incorrectly, send us an e-mail to: repliCATS-contact@unimelb.edu.au
How will a given research claim be replicated?
We cannot answer this question precisely. The selection and replication of claims for the SCORE program is being overseen by the Centre for Open Science, independently of the repliCATS project. See here for more details about this part of the SCORE program.

For SCORE, the intent of a direct replication is to follow the methods of the original study with a high degree of similarity, varying aspects only where there is a high degree of confidence that they are not relevant to the research claim being investigated. However, it is generally impossible to follow a study precisely, and the question as to which aspects matter is a judgement call.

Our best advice is to imagine what kinds of decisions you would face if you were asked to replicate this research claim, and then to consider the effects of making different choices for these decisions. This is one reason why we ask you to consider a set of 100 replications when making your assessment – even though they are 100 direct replications, each might be slightly different. You should consider the effect of these slight variations when making your estimate.

In some instances, a replication may not be able to collect new data, for example, if the claim relates to a specific historical event, like an election. In this case you should consider the different choices that could be made in analysing the data. Again, you should consider the effect of slight variations in these choices when making your estimate of replicability.

Platform troubleshooting

Can I use a tablet or handheld devide?
No, sorry! The online platform works best on a laptop or PC.
My browser doesn't seem to be working
We built the platform to be most compatible with Google Chrome. Safari seems to misbehave.
Do I need to save as I go?
Only if you want to. All responses have a save functionality at the question-level. It allows you to save as you go. This controls against losing information should your browser crash, or if you want to think about it/return to the question before you submit it to us.

$1000 monthly prizes

What are the rules for prizes?
See this post for all the rules, eligibility and how we pay out the monthly US$1000 in prizes.
When does each month/round begin?
We release new claims on the 17th of every month, starting from January 2020 to 17 June 2020.
Who are the past winners?
For a list of winners, see: https://replicats.research.unimelb.edu.au/2019/12/09/replicats-monthly-prizes-and-rules

About the project & SCORE program

What is “replication” as defined for this project?
Replication, along with many other related terms like reproducibility, are contested. That is, they have multiple meanings.

For this project, our working definition of a direct replication is a replication that follows the methods of the original study with a high degree of similarity, varying aspects only where there is a high degree of confidence that they are not relevant to the research claim. The aim of a direct replication is to improve confidence in the reliability and validity of an experimental finding by starting to account for things such as sampling error, measurement artefacts, and questionable research practices.
Does repliCATS stand for something?
Yes. The “CATS” in repliCATS is an acronym for Collaborative Assessment for Trustworthy Science.
Who is part of your research team?
We are an interdisciplinary research team based predominantly at the University of Melbourne. You can meet the research team here.
What are the aims of the repliCATS project?
We are developing and testing methods to elicit accurate predictions about the likely replicability of published research claims in the social sciences. As you may be aware, some large scale, crowdsourced replication projects have alerted us to the possibility that replication success rates may be lower than we once thought. Our project will assist with the development of efficient methods for critically evaluating the evidence base of social science research.
What is the IDEA protocol?
The IDEA protocol is a structured protocol for eliciting expert judgments based on the Delphi process. IDEA stands for Investigate, Discuss, Estimate, Aggregate.

Applying the IDEA protocol involves recruiting a diverse group of experts to answer questions with probabilistic or quantitative responses. Experts first investigate the questions and clarify meanings of terms, reducing variation caused by linguistic ambiguity. They provide their private, individual estimate, using a 3- or 4-step method (highest, lowest, best guess). The group’s private estimates are revealed; group members can then see how their estimates sit in relation to others. The group discusses the results, shares information and cross-examines reasoning and evidence. Group members individually provide a second and final private estimate. These second-round estimates are then combined using mathematical aggregation.

The strengths of the IDEA protocol in eliciting predictions of the likely replicability of research claims lies in the stepped, structured nature of the approach. The feedback and discussion components of the IDEA protocol both function to reduce overconfidence in estimates, which is a known limitation of expert elicitation methods. The discussion component of the IDEA protocol also allows experts to account for private information which could substantially alter the likely replicability assessment of a research claim.

This protocol, developed at the University of Melbourne, has been found to improve judgements under uncertainty. IDEA stands for “Investigate”, “Discuss”, “Estimate” and “Aggregate”, the four steps in the process of this elicitation.

More information on the IDEA protocol can be found here (external link to: Methods Blog).
Can I participate in this project?
Yes! We hope to collect judgements from a diverse range of participants in the following broad disciplines:
- business research
- criminology
- economics
- education
- political science
- psychology
- public administration
- marketing, and
- sociology.
If you are interested in participating, find out more about participating and signing upGet involved, or contact us at repliCATS-project@unimelb.edu.au to ask us for more information.
If I participate, what’s in it for me?
Your participation will help us to refine methods for predicting the replicability of social and behavioural science claims. Any data we collect could drastically change the way we think about published research evidence. For individuals participants, it also provides the opportunity to develop your skills, through peer interactions, and to become more critical consumers of the research literature.

Our first workshop was held in July 2019 in Rotterdam, with over 200 participants over two days. Our participants reported that they found the experience valuable and enjoyed thinking about replicability of published research evidence. Additionally, early career researchers said participating in the workshop improved their critical appraisal (or peer review) skills, and they enjoyed comparing their judgements against diverse individuals (from discipline to career stage) in their group.

How are the 3,000 research claims chosen?

criminology
economics
education
political science
psychology
public administration
marketing, and
sociology.

These claims will be drawn from the following journals.

Criminology		Marketing/Organisational Behaviour
Criminology Law and Human Behavior		Journal of Consumer Research Journal of Marketing Journal of Marketing Research Journal of Organizational Behavior Journal of the Academy of Marketing Science Organizational Behavior and Human Decision Processes

Economics		Political Science
American Economic Journal: Applied Economics American Economic Revie Econometrica Experimental Economics Journal of Finance Journal of Financial Economics Journal of Labor Economics Quarterly Journal of Economics Review of Financial Studies		American Political Science Review British Journal of Political Science Comparative Political Studies Journal of Conflict Resolution Journal of Experimental Political Science Journal of Political Economy World Development World Politics

Education		Psychology
American Educational Research Journal Computers and Education Contemporary Educational Psychology Educational Researcher Exceptional Children Journal of Educational Psychology Learning and Instruction		Child Development Clinical Psychological Science Cognition European Journal of Personality Evolution and Human Behavior Journal of Applied Psychology Journal of Consulting and Clinical Psychology Journal of Environmental Psychology Journal of Experimental Psychology: General Journal of Experimental Social Psychology Journal of Personality and Social Psychology Psychological Science

Health related		Public Administration
Health Psychology Psychological Medicine Social Science and Medicine		Journal of Public Administration Research and Theory Public Administration Review

Management		Sociology
Academy of Management Journal Journal of Business Research Journal of Management Leadership Quarterly Management Science Organization Science		American Journal of Sociology American Sociological Review Demography European Sociological Review Journal of Marriage and Family Social Forces

From which journals are the 3,000 research claims chosen?

The Center for Open Science (USA) are selecting the 3,000 research claims from the following journals.

Criminology		Marketing/Organisational Behaviour
Criminology Law and Human Behavior		Journal of Consumer Research Journal of Marketing Journal of Marketing Research Journal of Organizational Behavior Journal of the Academy of Marketing Science Organizational Behavior and Human Decision Processes

Economics		Political Science
American Economic Journal: Applied Economics American Economic Revie Econometrica Experimental Economics Journal of Finance Journal of Financial Economics Journal of Labor Economics Quarterly Journal of Economics Review of Financial Studies		American Political Science Review British Journal of Political Science Comparative Political Studies Journal of Conflict Resolution Journal of Experimental Political Science Journal of Political Economy World Development World Politics

Education		Psychology
American Educational Research Journal Computers and Education Contemporary Educational Psychology Educational Researcher Exceptional Children Journal of Educational Psychology Learning and Instruction		Child Development Clinical Psychological Science Cognition European Journal of Personality Evolution and Human Behavior Journal of Applied Psychology Journal of Consulting and Clinical Psychology Journal of Environmental Psychology Journal of Experimental Psychology: General Journal of Experimental Social Psychology Journal of Personality and Social Psychology Psychological Science

Health related		Public Administration
Health Psychology Psychological Medicine Social Science and Medicine		Journal of Public Administration Research and Theory Public Administration Review

Management		Sociology
Academy of Management Journal Journal of Business Research Journal of Management Leadership Quarterly Management Science Organization Science		American Journal of Sociology American Sociological Review Demography European Sociological Review Journal of Marriage and Family Social Forces

How can I get more information about this project?
You can express interest in assessing claims, or subscribe to our mailing list.

You can also follow us on twitter, @replicats.

Or, you can send us an e-mail at repliCATS-contact@unimelb.edu.au.

Collaborative Assessment for Trustworthy Science