Welcome to new workshop coordinators!

We’re really excited to have Ans Vercammen & Rania Poulis join the repliCATS project team to help coordinate our 2021 bushel workshops.

Ans VercammenAns Vercammen is trained as a research psychologist and completed a PhD in behavioural and cognitive neuroscience, studying the origins of auditory hallucinations. More recently, her interests have shifted away from neuropsychology to understanding why people (don’t) make pro-environmental choices and how to support environmental decision making at various scales. After completing an MSc in Conservation Science, she joined the Centre for Environmental Policy at Imperial College London, working with Mark Burgman on expert judgement. She recently joined the repliCATS Project to help coordinate Phase 2 workshops and conduct qualitative analyses as part of the Reasoning Team.


Rania PoulisRania Poulis has had a varied career beginning as a research assistant with a molecular biology/genomics background. She spent many years working to increase and improve gene-specific databases; this included a large amount of international conference organisation for the purpose of improving healthcare under the auspices of the Human Variome Project, an NGO of UNESCO. More recently Rania has been organising conferences for various scientific organisations and joined the team to help coordinate Phase 2 workshops and organise the AIMOS conference.

Express interest now open Jul-Sept bushel workshops.

Following our first bushel virtual workshop in partnership with SIPS, we can confirm workshop dates for July, August & September as follows:

  • 20-27 July – Economics, Management & Marketing papers (35-40 participants)
  • 24-31 August – Criminology, Political Science & Sociology (35-40 participants)
  • 21-28 September – Criminology, Political Science, Public administration (government and law) & Sociology (35-40 participants)

All bushel workshops will be virtual, participation is supported by an assessment grant of US$200, and will have a flexible assessment period.

For more information and to sign-up use the form below:

pre-SIPS repliCATS workshop – June 15-22

Hi folks, this post has all the info you need if you are participating in the pre-SIPS repliCATS bushel virtual workshop running from 15-22 June!

Here are links to all the information you’ll lead leading up to and during the week:

About the workshop

The workshop will run from 15-22 June (for your timezone), and during that time you’ll be asked to assess 2-3 bushel papers from psychology and adjacent disciplines. You’ll work in small groups of 4-6 people, each with a dedicated facilitator.

Here’s how it will work:

  • Before kick-off – you’ll get invited to the workshop slack space, receive your repliCATS platform sign-up details, be introduced to your group members & facilitator, and confirm your banking details so we can pay you the US$200 grants! (30-40 mins)
  • 15 June – workshop kick-off & introduction session* (45 mins) See below for details.
  • 15-20 June – complete your round 1 judgements (~2-2.5 hours, we don’t expect you to work on the weekends, but you have that extra time in case…)
  • 21-22 June – group discussion and submit your round 2 judgements (~1-1.5 hours). Group facilitators will contact you to schedule a synchronous virtual discussion at a time that suits your group on either Monday 21 June or Tuesday 22 June.
  • We’ll have a dedicated Slack space for everyone — there will be some friendly comps, some #random replipets fun.
  • Wrap-up & your feedback. We have something fun and new planned 🙂

*There will be two live sessions accommodating different time zones – you only need to turn up to one 🙂

Things to do before 07 June

  • Let us know if your availability changes at any time.
  • Complete consent & demographics. Before the workshop you must set up your account on the repliCATS platform, which includes completing a short demographics component and a quiz. This will take about 20-30 minutes, and must be completed in one sitting. The e-mail address you use on this form will be your repliCATS platform username too. At the end of the survey select “complete” from the options available to you.
  • Your banking details so we can pay you.
    • New participants: send us your first and last name as they appear on your bank account, and look for an e-mail from the University to set you up as a supplier.
    • Returning participants: you don’t need to do anything unless your bank details have changed.
  • Save the dates for the workshop introductions. These introductions will take you through everything you need to know! To accommodate different time zones, we are running two (identical) live webinars to kick-off the workshop. We will also record them for you to watch later.
Melbourne New Delhi Singapore London Washington DC Los Angeles Zoom links
Intro #1 Tue 15 Jun, 8am Tue 15 Jun, 3am Tue 15 Jun, 6am Mon 14 Jun, 11pm Mon 14 Jun, 6pm Mon 14 Jun, 3pm Check e-mail & slack
Intro #2 (repeat) Tue 15 Jun, 6pm Tue 15 Jun, 1.30pm Tue 15 Jun, 4pm Tue 15 Jun, 9am Tue 15 Jun, 4am Tue 15 Jun, 1am Check e-mail & slack

Find for the exact time for your city: https://www.timeanddate.com/worldclock/fixedform.html

  • Accept invitation into the slack workspace. For the week, we will have a dedicated slack workspace. Please try to use the same email address as the one you used for your consent & demographics so we know it’s you! Don’t want to use slack? See our faqs below.


👩🏻‍💻 repliCATS on call: Live Q&A troubleshooting assistance (AEST) on slack

We will have repliCATS staff members available during working hours throughout the week.

The slack channel is for communicating and troubleshooting (and fun) during the week, and is separate to the reasoning, discussion, and comments we ask you to provide while assessing claims on our repliCATS platform.

When you evaluate claims within the online platform you will have an avatar that protects your anonymity and the data are recorded for our project. On Slack, you are identifiable so you can get to know each other. We will not be recording your Slack conversations. We want to understand the way you think and reason about claims so we would really appreciate it if you keep any information about this on the repliCATS platform and for your discussion sessions.

If you have trouble with your Slack channel please contact us, repliCATS-project@unimelb.edu.au

#️⃣ twitter

You can tweet using #repliCATS2021 @repliCATS

Please don’t share estimates or claim information publicly, this helps us protect the integrity of the SCORE program.

😺 the repliCATS organising team

You can contact Ans, Rania and Mel via: repliCATS-project@unimelb.edu.au

Preparation & reference material

For workshop week, you will assess claims and follow an elicitation protocol termed the ‘IDEA protocol’. The following materials will help you to understand the protocol, why we are using it, and what we will ask of you during the workshop.

Intro video & audio files


The link to the platform is: https://replicats-bushel.eresearch.unimelb.edu.au/

The following documents are available via the Resources for participants page and may be useful if read before the introduction to the workshop. They can be easily accessed via the platform by clicking the resources button 🎓 :

  • repliCATS_BushelGuide_2021, plus a guide to round 1 and guide to round 2
  • Participant cheat sheet
  • Code of conduct

FAQs about the workshop

The following FAQs are specific to this virtual workshop. For general FAQs about answering claims, check the participant resources and the platform.

  • What if I don’t want to use Slack?
  • What can I discuss on Slack?
  • Who is in my group? And who is my facilitator?
  • How do I know when I have assessed all papers?
  • What if I become ill or cannot finish the workshop?
  • What if I can’t finish all papers on time?
  • What if I can’t attend my group’s discussion meeting?
  • I don’t like the claims assigned to my group.
  • I missed the introduction. Can I catch up?
  • Where or how can I give feedback?
  • What if I no longer want to take part?

What if I don’t want to use Slack?

We will be using slack to create a sense of fun and community as well as answering any questions you might have. However, if you do not want to use Slack you can send questions via email to repliCATS-project@unimelb.edu.au

You can also access the most important information you need to participate via the platform’s community page – there is a link to this web post. The community button (👥) is at the top right of your page once you have logged into the online platform. It is the 4th button along.

What can I discuss on Slack?

Discuss on Slack:

  • Questions about how to use the platform
  • Questions about the workshop generally
  • Clarifying the meaning of the questions

Save for the repliCATS platform & group discussion:

  • Descriptions of your reasons when evaluating a paper
  • Responses to other people’s justifications
  • Any information or discussion you might to have about a specific paper

How do I know when I have evaluated all the papers?

This should be really easy to track because each group of 4-6 people will be assigned 2-3 papers in total. We will ask you to do the following during the workshop:

Complete round 1 by 20 June
Complete discussion and round 2 by 22 June
We will contact you on Sunday, 20 June with an update on your progress. We will contact you again after the assessment period has ended confirming that you have reached your goal.

You can also check the status of a paper at any time in the platform by hovering over the little harvey balls on the top right of each paper. It will tell you how many people have completed round 1. See question below.

How do I know how many people are already in the paper?

The easiest way to tell is to hover over the little pie chart  (or harvey ball) on the homepage of the platform. It looks like this.

It will tell you how many people are assessing the paper, and which round they are in. The first number will always be the number of people in your group, the second number is how many people have saved a round 1 response, the third number how many have submitted a round 1 response, the fourth number is how many people have submitted a round 2 response.

What if I become ill?

We’re sorry to hear that, and hope you have mild symptoms. Please let us know, and we’ll work out a way for you to keep your grant that doesn’t involve you assessing papers while you are unwell!

See: answer below.

What if I can’t finish the papers on time?

Don’t worry – we know that things don’t always go according to plan.

As much as possible, we’d like for everyone to assess the papers assigned to you in the week. If you think you’ll struggle to make it, please reach out to us on repliCATS-contact@unimelb.edu.au and we’ll work something out with you.

What if I can’t attend my group’s discussion meeting?

We’d really like you to try and make your group’s discussion. If you can’t attend ––that is a shame because participants have consistently told us that discussing papers is one of the most fun and enriching parts of our workshops. However, we understand that when you are all working virtually/from home lots of unanticipated things might come up.

If you can’t make the scheduled discussion time, that’s OK. Let your facilitator know as soon as possible. You can still update your round 2 after 20 June — you will still have the benefit of reviewing your group’s responses and reasoning from round 1 before doing so.

I don’t like the papers assigned to my group.

We’ve tried our best to allocate papers that match the group’s expertise. We will not change the papers that have been assigned to your group except if you have been assigned a paper that you are working on a replication study for with TA1 or the Center for Open Science.

What if I no longer want to take part?

We’d be really sad to see you leave during assessment week, but we understand that things come up. Please let us know as soon as possible if you no longer want to take part, so we can offer the assessment grant to someone else on the waitlist.

I missed the introduction. Can I catch up?

Yes we will post the link to the recorded version in Slack and in the web post.

Where or how can I give feedback?

We’ll send you a link to an evaluation once the week is over. We’d love to know your thoughts and comments on the whole process.

Facilitator FAQs

Who’s in my group?

We have sent you an email with your group name and the link to the Facilitator Spreadsheet (also pinned to this post!). This contains all the information about your group and its members. If you can’t find the email, let us know via repliCATS-contact@unimelb.edu.au and we will resend you those details.

How do I contact my group members?

You will have a private group channel on Slack. If one or more of your group members does not want to use Slack, make sure you reach out to them via email to introduce yourself. They might also contact you via email.

I don’t know the answer to a question from one of my group members!

The #facilitator channel on Slack will be the hive mind of facilitators. Before you contact the repliCATS team, check the channel for info. It is likely that your question will have been posted before and answered. If not, ask us at repliCATS-contact@unimelb.edu.au.

When should I schedule my group’s discussion session?
People are busy, so try to settle on a date and time as soon as you can. We recommend scheduling the discussion for Monday 21 June or Tuesday 22 June to give people ample time to complete their Round 1 judgements.

What if someone can’t attend the discussion?

We encourage all group members to attend the discussion, but it might not be possible to find a date and time that suits everyone, and sometimes people’s plans change at the last minute (e.g. they might have responsibilities as a carer). It is helpful for us to know how many people attended the discussion, so please log attendance on the Facilitator Spreadsheet (pinned to the post in the slack channel).

I have never facilitated a discussion before – what do I do?

Your main role is to ensure that everyone gets a chance to talk and to keep the discussion focused on exploring the reasoning that underpins Round 1 judgements. For more detailed guidance, review the facilitator training (a recording can be found here) and the Guidelines for Facilitators (pinned to the post in the slack channel)

When does Round 2 start?

Whenever your group has discussed a paper, participants can start updating their estimates. Whether you want to do this in real-time, i.e. as you are discussing the questions, or after the discussion is finished, is up to you.

What happens at the end of the workshop?

Please be sure to thank your group on behalf of the repliCATS team. Stay tuned for a wrap-up event. It will be different and fun, but as usual, there will be awards! Remind your participants that it might be up to 6 weeks for their assessment grants to be processed.

A SCORE of pre-prints.

In phase 1, we crowdsourced judgements of replicability for 3000 research claims drawn from peer-reviewed journals across eight social and behavioural science disciplines. The following thread unrolls a series of pre-prints now available describing the overall SCORE collaborative in phase 1 (involving five research teams from around the world), as well as a number of articles from the repliCATS project team, including:

First 2021 workshop announced!

repliCATS workshops will be back in June! We are kicking off our phase 2 research with a pre-SIPS repliCATS virtual workshop from 15-22 June 2021.

What’s new?
In 2021, the repliCATS workshops will focus on evaluating the credibility of entire published research papers (or what we are calling “bushel papers”) in the social and behavioural sciences literature. For the pre-SIPS workshop, the papers you evaluate will be drawn from psychology and psychology-adjacent fields.

However, our workshops will remain familiar to returning participants. Working in small groups of 4-6, you will be asked to evaluate 2-3 papers. Each group will have its own facilitator, who will guide you through a structured elicitation protocol.

For more details and to express interest in participating in this workshop, use the following form:


SCORE program renewed for phase 2

We’re excited to announce that DARPA has renewed the SCORE program—and the repliCATS project—for a second phase!

What does that mean? It means your predictions met accuracy thresholds! And we’ll return in 2021. In phase 2, we’ll evaluate a fresh set of claims for likely replicability as well as ask new questions, for example, about validity and generalisability. We’ll continue our work developing the repliCATS protocol for alternative rapid peer review, and peer review training using the opportunity for feedback & calibration that our protocol provides.
All of our achievements to date are attributable to the 550+ participants who have collaborated with us so far. We hope you’ll continue to join our workshops, and earn our badges, and keep us accountable.

For more information about the wider SCORE program, see: https://www.darpa.mil/program/systematizing-confidence-in-open-research-and-evidence

Watch 📺 Metascience should not be defined by its methods

Join our CI Fiona Fidler & philosopher of science Rachel Brown for a webinar on “Metascience should not be defined by its methods.

Metascience, or metaresearch, is a field of research that has grown out of the replication crisis. Amongst other things, metascience evaluates and monitors open science initiatives and other interventions to improve scientific practices and cultures. It studies the impact of institutional reward structures, performance metrics, incentives, and methods of scientific resource allocation. It also evaluates peer review practices—both current, and alternative future proposals. As a field, it is (unfortunately) often defined as “the scientific study of science itself” or “the science of doing science”. In this talk I’ll explain why metascience shouldn’t be defined by appeal to a monolithic scientific method, and discuss other ways of conceptualizing what this new epistemic community does.

The webinar is now available to watch here: https://biology.anu.edu.au/news-events/events/ee-webinar-metascience-should-not-be-defined-its-methods

Closed: Express interest in assessing COVID-19 research claims

The repliCATS project — and the wider SCORE program — has been expanded by DARPA to include 100 COVID-19 research claims from the social and behavioural sciences.

Governments around the world may rely on social and behavioural science research to understand crises and inform policies. Inclusion of COVID claims will ensure that the repliCATS project is tested on a range of social and behavioural science or SBS research areas, making it uniquely tailored for the critical evaluation of a broad spectrum of research claims.

The repliCATS project will run a virtual worksop from 2-8 September to assess these COVID-19 claims, and are looking for 100 participants.

Are you new to the repliCATS project? If so, you can watch our chief investigator, Prof Fiona Fidler introduce the project, or read more about what it means to assess research claims.

Express interest in workshop (closed) by Friday, 14 August

Expressions of interest in participating in our workshop are now open, but places are capped at 100 participants.

**UPDATE: EOIs closed on 14 August. Thank you so much for expressing interest. You will have received an e-mail letting you know if you have been confirmed for the workshop or if you are on a waitlist. Rolling offers were made on a first-come basis. If you have any questions, please contact us on repliCATS-project@unimelb.edu.au**

The key details again (in case you missed them).

  • There are 100 x USD200 (AUD 280) assessment grants available for eligible participants.
  • You need to be available between 2-8 September, for ~ 6 hours over the week.
  • *Each participant will be asked to assess ten claims.
  • You can assess claims when it suits you.
  • You’ll work in groups of 3-6 other participants, using our online platform.
  • If this is your first time assessing claims, don’t worry. We’ll run an intro session to help you get started, and be available to help you over the week.
  • *If you have already expressed interest before 03 August, you do not need to re-apply. We will contact you shortly with additional details about the workshop, including assessment grant eligibility.

Form not working? Click here.

Number of posts found: 31