Get involved

Participating in our project means making judgements about the credibility of a published research claim.

We are currently collaborating with the Center for Open Science, extending the work we started with the SCORE program. With new funding from the Robert Wood Johnson Foundation(US), we are kicking off a new program, SMART or Scaling Machines Assessments of Research Trustworthiness. 

The first part of this program is the SMART – preprints project where we will collaborate with preprint authors to evaluate the reliability of preprints in psychology, sociology & adjacent disciplines!

Workshop schedule for 2023 & 2024

  • Pilot workshop – pre-AIMOS2023 conference (Brisbane), Monday, 20 November 2023 from 11am – 4.30pm. [Assessment grants of AU$150 available, priority given to AIMOS registrants.]
  • Workshop 1 – SPSP2024 convention (San Diego), 8-10 Febraury 2024, details TBC.
  • Workshop 2 – virtual workshop, assessment period between Thursday 18 April – Tuesday 23 April 2024.
  • Workshop 3 – pre-SIPS2024 (Nairobi), details TBC.
  • Workshop 4 – virtual workshop, assessment period between Thursday 15 August – Tuesday 20 August 2024.

To express interest in any or all of these workshops, use this form.

 

What does “participating” mean exactly?

We will ask you to use our online repliCATS platform to make a series of judgements about a published research claim, for example, how well you understand the claim, how replicable you think the claim is, and if the statistical tools and conclusions drawn by the authors are valid.

What won’t change is our approach. That is, we don’t ask you to do this alone. Our method (the IDEA protocol) involves structured group discussion – each claim is assessed by 3-5 other people, and you get to see what others in your group say before submitting your final judgement.

For the SMART preprints project:

  • Participants will be eligible for US$150-$200 assessment grants.
  • Express interest in participating in phase 2, and we’ll let you know when we open sign-ups for workshops.
  • To express interest for upcoming workshops, use this form:

Why get involved?

Over the life of the SCORE program we achieved something extraordinary! We had over 1000 participants from around the world contribute to evaluating the 4000 articles and 100 COVID-19 preprints. What we learned from the SCORE program was that folks were excited to be a part the largest effort to evaluate reliability in the social & behavioural sciences!

But also, participants get to:

  • improve their peer-review & error detection skills
  • calibrate judgements & reasoning against their peers

and that our process is straight-up fun!

Who can assess claims? Every participant counts – don’t worry about being an expert, we need diverse views

Our method – the IDEA protocol – harnesses the power of structured group discussion in evaluating the credibility of published research. We have built a custom cloud-based platform to gather your evaluations. What we ask you to do is to evaluate the credibility of a claim, that is we ask you to read a paper and evaluate a set of credibility signals for that paper, including comprehensibility, validity, robustness and replicability.

Part of the scope of the repliCATS project is to examine the markers of expertise (e.g. education, experience, domain knowledge), and the role they may play in making good judgements about the likelihood a research claim will replicate.

This means an eligible research participant for our project is someone who has completed or is completing a relevant undergraduate degree, and is over 18 years of age. And, importantly, is interested in making judgements about published research.

If you would like more information, you can:

SCORE program (2019-2022)

Data collection for the repliCATS project under the SCORE program is now complete. We hope to share results with you soon, and we will continue to pre-print and publish our work as we make progress.

–

–

Just want to stay up-to-date on the project? Subscribe to our newsletter

We have a quarterly newsletter we send out about our project. By subscribing you’ll get a short, snappy newsletter letting you know what we’ve been up to, and what’s happening with the repliCATS project.

You just sign-up using this form (we won’t spam you)

 

Privacy Collection Notice – the repliCATS project.

SMART preprints human ethics application ID: 2023-27416-44991-3 [Robert Wood John Foundation grant].

SCORE program human ethics application ID: 1853445.1 [project complete].

The information on this form is being collected by the repliCATS project, a research group at the University of Melbourne. You can contact us at repliCATS-contact@unimelb.edu.au.

The information you provide will be used to communicate with you about the repliCATS project. The information will be used by authorised staff for the purpose for which it was collected, and will be protected against unauthorised access and use.

You may access any personal information you have provided to the University by contacting us at repliCATS-contact@unimelb.edu.au. The University of Melbourne is committed to protecting personal information provided by you in accordance with the Privacy and Data Protection Act 2014 (Vic). All information collected by the University is governed by the University’s Privacy Policy. For further information about how the University deals with personal information, please refer to the University’s Privacy Policy or contact the University’s Privacy Officer at privacy-officer@unimelb.edu.au