September workshop – 21-28 Sept 2021

Hi folks, this post has all the info you need if you are participating in the repliCATS September workshop running from 21-28 September!

Here are links to all the information you’ll lead leading up to and during the week:

About the workshop

The workshop will run from 21-28 September (for your timezone), and during that time you’ll be asked to assess 2-3 bushel papers from political science, sociology and adjacent disciplines. You’ll work in small groups of 4-6 people, each with a dedicated facilitator.

Here’s how it will work:

  • Before kick-off – you’ll get invited to the workshop slack space, receive your repliCATS platform sign-up details, be introduced to your group members & facilitator, and confirm your banking details so we can pay you the US$200 grants! (30-40 mins)
  • 21 September – workshop kick-off & introduction session* (45 mins). See below for details.
  • 21-26 September – complete your round 1 judgements (~2-2.5 hours, we don’t expect you to work on the weekends, but you have that extra time in case…)
  • 27 & 28 September – group discussion and submit your round 2 judgements (~1-1.5 hours). Group facilitators will contact you to schedule a synchronous virtual discussion at a time that suits your group on either the Monday or Tuesday.
  • We’ll have a dedicated Slack space for everyone — there will be some friendly comps, some #random replipets fun.
  • Wrap-up & your feedback.

*There will be two live sessions accommodating different time zones – you only need to turn up to one 🙂

Things to do before the workshop

  • Let us know if your availability changes at any time.
  • Your banking details so we can pay you.
    • New participants: send us your first and last name as they appear on your bank account, and look for an e-mail from the University to set you up as a supplier.
    • Returning participants: you don’t need to do anything unless your bank details have changed.
  • Save the dates for the workshop introductions. These introductions will take you through everything you need to know! If you are a return participant, you don’t have to attend the workshop intro … unless you want a refresher! We will also record the intro and post it in the Slack channel for you to watch later.
  • Accept invitation into the slack workspace. For the week, we will have a dedicated slack workspace. Please try to use the same email address as the one you used for your consent & demographics so we know it’s you! Don’t want to use slack? See our faqs below.

Communication

👩🏻‍💻 repliCATS on call: Live Q&A troubleshooting assistance (AEST) on slack

We will have repliCATS staff members available during working hours throughout the week.

The slack channel is for communicating and troubleshooting (and fun) during the week, and is separate to the reasoning, discussion, and comments we ask you to provide while assessing claims on our repliCATS platform.

When you evaluate claims within the online platform you will have an avatar that protects your anonymity and the data are recorded for our project. On Slack, you are identifiable so you can get to know each other. We will not be recording your Slack conversations. We want to understand the way you think and reason about claims so we would really appreciate it if you keep any information about this on the repliCATS platform and for your discussion sessions.

If you have trouble with your Slack channel please contact us, repliCATS-project@unimelb.edu.au

#️⃣ twitter

You can tweet using #repliCATS2021 @repliCATS

Please don’t share estimates or claim information publicly, this helps us protect the integrity of the SCORE program.

😺 the repliCATS organising team

You can contact Ans, Rania and Mel via: repliCATS-project@unimelb.edu.au

Preparation & reference material

For workshop week, you will assess claims and follow an elicitation protocol termed the ‘IDEA protocol’. The following materials will help you to understand the protocol, why we are using it, and what we will ask of you during the workshop. Everything you need can be found via the Resources for participants page.

The link to the platform is: https://replicats-bushel.eresearch.unimelb.edu.au/

The following documents are available via the Resources for participants page and may be useful if read before the introduction to the workshop. They can be easily accessed via the platform by clicking the resources button 🎓 :

  • repliCATS_BushelGuide_2021, plus a guide to round 1 and guide to round 2
  • Participant cheat sheet
  • Code of conduct

FAQs about the workshop

The following FAQs are specific to this virtual workshop. For general FAQs about answering claims, check the participant resources and the platform.

  • What if I don’t want to use Slack?
  • What can I discuss on Slack?
  • Who is in my group? And who is my facilitator?
  • How do I know when I have assessed all papers?
  • What if I become ill or cannot finish the workshop?
  • What if I can’t finish all papers on time?
  • What if I can’t attend my group’s discussion meeting?
  • I don’t like the claims assigned to my group.
  • I missed the introduction. Can I catch up?
  • Where or how can I give feedback?
  • What if I no longer want to take part?

What if I don’t want to use Slack?

We will be using slack to create a sense of fun and community as well as answering any questions you might have. However, if you do not want to use Slack you can send questions via email to repliCATS-project@unimelb.edu.au

You can also access the most important information you need to participate via the platform’s community page – there is a link to this web post. The community button (👥) is at the top right of your page once you have logged into the online platform. It is the 4th button along.

What can I discuss on Slack?

Discuss on Slack:

  • Questions about how to use the platform
  • Questions about the workshop generally
  • Clarifying the meaning of the questions

Save for the repliCATS platform & group discussion:

  • Descriptions of your reasons when evaluating a paper
  • Responses to other people’s justifications
  • Any information or discussion you might to have about a specific paper

How do I know when I have evaluated all the papers?

This should be really easy to track because each group of 4-6 people will be assigned 2-3 papers in total. We will ask you to do the following during the workshop:

  • Complete round 1 within the first six days
  • Complete discussion and round 2 in the last two days of the assessment period
  • Your facilitator will contact you with an update on your progress. We will contact you again after the assessment period has ended confirming that you have reached your goal.

You can also check the status of a paper at any time in the platform by hovering over the little harvey balls on the top right of each paper. It will tell you how many people have completed round 1. See question below.

How do I know how many people are already in the paper?

The easiest way to tell is to hover over the little pie chart  (or harvey ball) on the homepage of the platform. It looks like this.

It will tell you how many people are assessing the paper, and which round they are in. The first number will always be the number of people in your group, the second number is how many people have saved a round 1 response, the third number how many have submitted a round 1 response, the fourth number is how many people have submitted a round 2 response.

What if I become ill?

We’re sorry to hear that, and hope you have mild symptoms. Please let us know, and we’ll work out a way for you to keep your grant that doesn’t involve you assessing papers while you are unwell!

See: answer below.

What if I can’t finish the papers on time?

Don’t worry – we know that things don’t always go according to plan.

As much as possible, we’d like for everyone to assess the papers assigned to you in the week. If you think you’ll struggle to make it, please reach out to us on repliCATS-contact@unimelb.edu.au and we’ll work something out with you.

What if I can’t attend my group’s discussion meeting?

We’d really like you to try and make your group’s discussion. If you can’t attend ––that is a shame because participants have consistently told us that discussing papers is one of the most fun and enriching parts of our workshops. However, we understand that when you are all working virtually/from home lots of unanticipated things might come up.

If you can’t make the scheduled discussion time, that’s OK. Let your facilitator know as soon as possible. You can still update your round 2 after the end of the workshop — you will still have the benefit of reviewing your group’s responses and reasoning from round 1 before doing so.

I don’t like the papers assigned to my group.

We’ve tried our best to allocate papers that match the group’s expertise. We will not change the papers that have been assigned to your group except if you have been assigned a paper that you are working on a replication study for with TA1 or the Center for Open Science.

What if I no longer want to take part?

We’d be really sad to see you leave during assessment week, but we understand that things come up. Please let us know as soon as possible if you no longer want to take part, so we can offer the assessment grant to someone else on the waitlist.

I missed the introduction. Can I catch up?

Yes we will post the link to the recorded version in Slack and in the web post.

Where or how can I give feedback?

We’ll send you a link to an evaluation once the week is over. We’d love to know your thoughts and comments on the whole process.

Facilitator FAQs

Who’s in my group?

We have sent you an email with your group name and the link to the Facilitator Spreadsheet (also pinned to this post!). This contains all the information about your group and its members. If you can’t find the email, let us know via repliCATS-contact@unimelb.edu.au and we will resend you those details.

How do I contact my group members?

You will have a private group channel on Slack. If one or more of your group members does not want to use Slack, make sure you reach out to them via email to introduce yourself. They might also contact you via email.

I don’t know the answer to a question from one of my group members!

The #facilitator channel on Slack will be the hive mind of facilitators. Before you contact the repliCATS team, check the channel for info. It is likely that your question will have been posted before and answered. If not, ask us at repliCATS-contact@unimelb.edu.au.

When should I schedule my group’s discussion session?
People are busy, so try to settle on a date and time as soon as you can. We recommend scheduling the discussion for Monday 26 July or Tuesday 27 July to give people ample time to complete their Round 1 judgements.

What if someone can’t attend the discussion?

We encourage all group members to attend the discussion, but it might not be possible to find a date and time that suits everyone, and sometimes people’s plans change at the last minute (e.g. they might have responsibilities as a carer). It is helpful for us to know how many people attended the discussion, so please log attendance on the Facilitator Spreadsheet (pinned to the post in the slack channel).

I have never facilitated a discussion before – what do I do?

Your main role is to ensure that everyone gets a chance to talk and to keep the discussion focused on exploring the reasoning that underpins Round 1 judgements. For more detailed guidance, review the facilitator training (a recording can be found here) and the Guidelines for Facilitators (pinned to the post in the slack channel)

When does Round 2 start?

Whenever your group has discussed a paper, participants can start updating their estimates. Whether you want to do this in real-time, i.e. as you are discussing the questions, or after the discussion is finished, is up to you.

What happens at the end of the workshop?

Please be sure to thank your group on behalf of the repliCATS team. Stay tuned for a wrap-up event. It will be different and fun, but as usual, there will be awards! Remind your participants that it might be up to 6 weeks for their assessment grants to be processed.


Sign-up for Sept, Oct & Nov bushel workshops.

You can now sign-up for one of our workshops! Jump straight to form.

  • 21-28 September – Criminology, Political Science, Public administration (government and law) & Sociology (35-40 participants, limited spots available)
    • Looking for participants in EU, North America time zones
  • 19-26 October – Economics, Management & Marketing (35-40 participants)
    • In partnership with UCMeta in Canterbury (NZ). This workshop will be open to Australia, New Zealand and all Asian countries.
  • 18-25 November – Criminology, Economics, Education, Government & law, Management, Marketing Political Science & Sociology (55-70 participants)
    • In partnership with aimos2021 conference. This workshop will be open to a global participant pool.

New to workshops? Here’s what to expect.

A bit about the repliCATS project
Based at the University of Melbourne, the repliCATS project team are part of a wider program called SCORE funded by DARPA. We are excited about reimagining peer review as a structured deliberation process. We’re testing this by crowdsourcing expert judgements about the credibility of published research in eight social science disciplines – criminology, education, economics, marketing, management, political science, psychology, public administration, and sociology.

repliCATS workshops – get paid to do post-publication peer review!
In workshops you will be asked to evaluate the credibility of two to three published papers within your domain expertise. We use a structured group deliberation approach called the IDEA protocol on a custom-built web-based platform which means you can participate from anywhere in the world! Working in small groups, you will be asked to evaluate papers across a number of credibility signals – from its comprehensibility, to its replicability, robustness and transparency.

Participants will first make private judgements about the paper, and then get to review and discuss their group’s responses before submitting their final judgements. For these workshops, we’ll group you by compatible time zones, and each group will have its own facilitator who will guide you through a short virtual discussion.

All participants are eligible for a US$200 assessment grant.

All bushel workshops are virtual, and your participation is supported by an assessment grant of US$200.

The workshops run for a period of a week (for your timezone), and during that time you’ll be asked to assess 2-3 bushel papers. You’ll work in small groups of 4-6 people, each with a dedicated facilitator.

Here’s how it will work:

  • Before kick-off – you’ll get invited to the workshop slack space, receive your repliCATS platform sign-up details, be introduced to your group members & facilitator, and confirm your banking details so we can pay you the US$200 grants! (30-40 mins)
  • Day 1 – workshop kick-off & introduction session to get your started (45 mins, recorded)
  • Day 1-5 – complete your round 1 judgements (~2-2.5 hours)
  • Day 6 or 7 – group discussion and submit your round 2 judgements (~1-1.5 hours). Group facilitators will contact you to schedule a synchronous virtual discussion at a time that suits your group.

For more information and to sign-up use the form below:


August workshop – 24-31 August

Hi folks, this post has all the info you need if you are participating in the repliCATS August workshop running from 24-31 August!

Here are links to all the information you’ll lead leading up to and during the week:

About the workshop

The workshop will run from 24-31 August (for your timezone), and during that time you’ll be asked to assess 2-3 bushel papers from political science, sociology and adjacent disciplines. You’ll work in small groups of 4-6 people, each with a dedicated facilitator.

Here’s how it will work:

  • Before kick-off – you’ll get invited to the workshop slack space, receive your repliCATS platform sign-up details, be introduced to your group members & facilitator, and confirm your banking details so we can pay you the US$200 grants! (30-40 mins)
  • 24 August – workshop kick-off & introduction session* (45 mins). See below for details.
  • 24-29 August – complete your round 1 judgements (~2-2.5 hours, we don’t expect you to work on the weekends, but you have that extra time in case…)
  • 30 & 31 August – group discussion and submit your round 2 judgements (~1-1.5 hours). Group facilitators will contact you to schedule a synchronous virtual discussion at a time that suits your group on either Monday 30 August or Tuesday 31 August.
  • We’ll have a dedicated Slack space for everyone — there will be some friendly comps, some #random replipets fun.
  • Wrap-up & your feedback.

*There will be two live sessions accommodating different time zones – you only need to turn up to one 🙂

Things to do before the workshop

  • Let us know if your availability changes at any time.
  • Your banking details so we can pay you.
    • New participants: send us your first and last name as they appear on your bank account, and look for an e-mail from the University to set you up as a supplier.
    • Returning participants: you don’t need to do anything unless your bank details have changed.
  • Save the dates for the workshop introductions. These introductions will take you through everything you need to know! To accommodate different time zones, we are running two (identical) live webinars to kick-off the workshop. We will also record them for you to watch later.
Melbourne New Delhi Singapore London Washington DC Los Angeles Zoom links
Intro #1 tbc Check e-mail & slack
Intro #2 (repeat) tbc Check e-mail & slack

Find for the exact time for your city: https://www.timeanddate.com/worldclock/fixedform.html

  • Accept invitation into the slack workspace. For the week, we will have a dedicated slack workspace. Please try to use the same email address as the one you used for your consent & demographics so we know it’s you! Don’t want to use slack? See our faqs below.

Communication

👩🏻‍💻 repliCATS on call: Live Q&A troubleshooting assistance (AEST) on slack

We will have repliCATS staff members available during working hours throughout the week.

The slack channel is for communicating and troubleshooting (and fun) during the week, and is separate to the reasoning, discussion, and comments we ask you to provide while assessing claims on our repliCATS platform.

When you evaluate claims within the online platform you will have an avatar that protects your anonymity and the data are recorded for our project. On Slack, you are identifiable so you can get to know each other. We will not be recording your Slack conversations. We want to understand the way you think and reason about claims so we would really appreciate it if you keep any information about this on the repliCATS platform and for your discussion sessions.

If you have trouble with your Slack channel please contact us, repliCATS-project@unimelb.edu.au

#️⃣ twitter

You can tweet using #repliCATS2021 @repliCATS

Please don’t share estimates or claim information publicly, this helps us protect the integrity of the SCORE program.

😺 the repliCATS organising team

You can contact Ans, Rania and Mel via: repliCATS-project@unimelb.edu.au

Preparation & reference material

For workshop week, you will assess claims and follow an elicitation protocol termed the ‘IDEA protocol’. The following materials will help you to understand the protocol, why we are using it, and what we will ask of you during the workshop.

Intro video & audio files (NB. this is from the June workshop. We will update the video on 20 July!)

 

The link to the platform is: https://replicats-bushel.eresearch.unimelb.edu.au/

The following documents are available via the Resources for participants page and may be useful if read before the introduction to the workshop. They can be easily accessed via the platform by clicking the resources button 🎓 :

  • repliCATS_BushelGuide_2021, plus a guide to round 1 and guide to round 2
  • Participant cheat sheet
  • Code of conduct

FAQs about the workshop

The following FAQs are specific to this virtual workshop. For general FAQs about answering claims, check the participant resources and the platform.

  • What if I don’t want to use Slack?
  • What can I discuss on Slack?
  • Who is in my group? And who is my facilitator?
  • How do I know when I have assessed all papers?
  • What if I become ill or cannot finish the workshop?
  • What if I can’t finish all papers on time?
  • What if I can’t attend my group’s discussion meeting?
  • I don’t like the claims assigned to my group.
  • I missed the introduction. Can I catch up?
  • Where or how can I give feedback?
  • What if I no longer want to take part?

What if I don’t want to use Slack?

We will be using slack to create a sense of fun and community as well as answering any questions you might have. However, if you do not want to use Slack you can send questions via email to repliCATS-project@unimelb.edu.au

You can also access the most important information you need to participate via the platform’s community page – there is a link to this web post. The community button (👥) is at the top right of your page once you have logged into the online platform. It is the 4th button along.

What can I discuss on Slack?

Discuss on Slack:

  • Questions about how to use the platform
  • Questions about the workshop generally
  • Clarifying the meaning of the questions

Save for the repliCATS platform & group discussion:

  • Descriptions of your reasons when evaluating a paper
  • Responses to other people’s justifications
  • Any information or discussion you might to have about a specific paper

How do I know when I have evaluated all the papers?

This should be really easy to track because each group of 4-6 people will be assigned 2-3 papers in total. We will ask you to do the following during the workshop:

  • Complete round 1 by 25 July
  • Complete discussion and round 2 by 27 July
  • We will contact you on Sunday with an update on your progress. We will contact you again after the assessment period has ended confirming that you have reached your goal.

You can also check the status of a paper at any time in the platform by hovering over the little harvey balls on the top right of each paper. It will tell you how many people have completed round 1. See question below.

How do I know how many people are already in the paper?

The easiest way to tell is to hover over the little pie chart  (or harvey ball) on the homepage of the platform. It looks like this.

It will tell you how many people are assessing the paper, and which round they are in. The first number will always be the number of people in your group, the second number is how many people have saved a round 1 response, the third number how many have submitted a round 1 response, the fourth number is how many people have submitted a round 2 response.

What if I become ill?

We’re sorry to hear that, and hope you have mild symptoms. Please let us know, and we’ll work out a way for you to keep your grant that doesn’t involve you assessing papers while you are unwell!

See: answer below.

What if I can’t finish the papers on time?

Don’t worry – we know that things don’t always go according to plan.

As much as possible, we’d like for everyone to assess the papers assigned to you in the week. If you think you’ll struggle to make it, please reach out to us on repliCATS-contact@unimelb.edu.au and we’ll work something out with you.

What if I can’t attend my group’s discussion meeting?

We’d really like you to try and make your group’s discussion. If you can’t attend ––that is a shame because participants have consistently told us that discussing papers is one of the most fun and enriching parts of our workshops. However, we understand that when you are all working virtually/from home lots of unanticipated things might come up.

If you can’t make the scheduled discussion time, that’s OK. Let your facilitator know as soon as possible. You can still update your round 2 after the end of the workshop — you will still have the benefit of reviewing your group’s responses and reasoning from round 1 before doing so.

I don’t like the papers assigned to my group.

We’ve tried our best to allocate papers that match the group’s expertise. We will not change the papers that have been assigned to your group except if you have been assigned a paper that you are working on a replication study for with TA1 or the Center for Open Science.

What if I no longer want to take part?

We’d be really sad to see you leave during assessment week, but we understand that things come up. Please let us know as soon as possible if you no longer want to take part, so we can offer the assessment grant to someone else on the waitlist.

I missed the introduction. Can I catch up?

Yes we will post the link to the recorded version in Slack and in the web post.

Where or how can I give feedback?

We’ll send you a link to an evaluation once the week is over. We’d love to know your thoughts and comments on the whole process.

Facilitator FAQs

Who’s in my group?

We have sent you an email with your group name and the link to the Facilitator Spreadsheet (also pinned to this post!). This contains all the information about your group and its members. If you can’t find the email, let us know via repliCATS-contact@unimelb.edu.au and we will resend you those details.

How do I contact my group members?

You will have a private group channel on Slack. If one or more of your group members does not want to use Slack, make sure you reach out to them via email to introduce yourself. They might also contact you via email.

I don’t know the answer to a question from one of my group members!

The #facilitator channel on Slack will be the hive mind of facilitators. Before you contact the repliCATS team, check the channel for info. It is likely that your question will have been posted before and answered. If not, ask us at repliCATS-contact@unimelb.edu.au.

When should I schedule my group’s discussion session?
People are busy, so try to settle on a date and time as soon as you can. We recommend scheduling the discussion for Monday 26 July or Tuesday 27 July to give people ample time to complete their Round 1 judgements.

What if someone can’t attend the discussion?

We encourage all group members to attend the discussion, but it might not be possible to find a date and time that suits everyone, and sometimes people’s plans change at the last minute (e.g. they might have responsibilities as a carer). It is helpful for us to know how many people attended the discussion, so please log attendance on the Facilitator Spreadsheet (pinned to the post in the slack channel).

I have never facilitated a discussion before – what do I do?

Your main role is to ensure that everyone gets a chance to talk and to keep the discussion focused on exploring the reasoning that underpins Round 1 judgements. For more detailed guidance, review the facilitator training (a recording can be found here) and the Guidelines for Facilitators (pinned to the post in the slack channel)

When does Round 2 start?

Whenever your group has discussed a paper, participants can start updating their estimates. Whether you want to do this in real-time, i.e. as you are discussing the questions, or after the discussion is finished, is up to you.

What happens at the end of the workshop?

Please be sure to thank your group on behalf of the repliCATS team. Stay tuned for a wrap-up event. It will be different and fun, but as usual, there will be awards! Remind your participants that it might be up to 6 weeks for their assessment grants to be processed.


July workshop info – 20-27 July

Hi folks, this post has all the info you need if you are participating in the repliCATS July workshop running from 20-27 July!

Here are links to all the information you’ll lead leading up to and during the week:

About the workshop

The workshop will run from 20-27 July (for your timezone), and during that time you’ll be asked to assess 2-3 bushel papers from economics, management, marketing and adjacent disciplines. You’ll work in small groups of 4-6 people, each with a dedicated facilitator.

Here’s how it will work:

  • Before kick-off – you’ll get invited to the workshop slack space, receive your repliCATS platform sign-up details, be introduced to your group members & facilitator, and confirm your banking details so we can pay you the US$200 grants! (30-40 mins)
  • 20 July – workshop kick-off & introduction session* (45 mins) See below for details.
  • 20-25 July – complete your round 1 judgements (~2-2.5 hours, we don’t expect you to work on the weekends, but you have that extra time in case…)
  • 26-27 July – group discussion and submit your round 2 judgements (~1-1.5 hours). Group facilitators will contact you to schedule a synchronous virtual discussion at a time that suits your group on either Monday 26 July or Tuesday 27 July.
  • We’ll have a dedicated Slack space for everyone — there will be some friendly comps, some #random replipets fun.
  • Wrap-up & your feedback.

*There will be two live sessions accommodating different time zones – you only need to turn up to one 🙂

Things to do before the workshop

  • Let us know if your availability changes at any time.
  • Your banking details so we can pay you.
    • New participants: send us your first and last name as they appear on your bank account, and look for an e-mail from the University to set you up as a supplier.
    • Returning participants: you don’t need to do anything unless your bank details have changed.
  • Save the dates for the workshop introductions. These introductions will take you through everything you need to know! To accommodate different time zones, we are running two (identical) live webinars to kick-off the workshop. We will also record them for you to watch later.
Melbourne New Delhi Singapore London Washington DC Los Angeles Zoom links
Intro #1 Tue 20 Jul, 8am Tue 20 Jul, 3am Tue 20 Jul, 6am Mon 19 Jul, 11pm Mon 19 Jul, 6pm Mon 19 Jul, 3pm Check e-mail & slack
Intro #2 (repeat) Tue 20 Jul, 6pm Tue 20 Jul, 1.30pm Tue 20 Jul, 4pm Tue 20 Jul, 9am Tue 20 Jul, 4am Tue 20 Jul, 1am Check e-mail & slack

Find for the exact time for your city: https://www.timeanddate.com/worldclock/fixedform.html

  • Accept invitation into the slack workspace. For the week, we will have a dedicated slack workspace. Please try to use the same email address as the one you used for your consent & demographics so we know it’s you! Don’t want to use slack? See our faqs below.

Communication

👩🏻‍💻 repliCATS on call: Live Q&A troubleshooting assistance (AEST) on slack

We will have repliCATS staff members available during working hours throughout the week.

The slack channel is for communicating and troubleshooting (and fun) during the week, and is separate to the reasoning, discussion, and comments we ask you to provide while assessing claims on our repliCATS platform.

When you evaluate claims within the online platform you will have an avatar that protects your anonymity and the data are recorded for our project. On Slack, you are identifiable so you can get to know each other. We will not be recording your Slack conversations. We want to understand the way you think and reason about claims so we would really appreciate it if you keep any information about this on the repliCATS platform and for your discussion sessions.

If you have trouble with your Slack channel please contact us, repliCATS-project@unimelb.edu.au

#️⃣ twitter

You can tweet using #repliCATS2021 @repliCATS

Please don’t share estimates or claim information publicly, this helps us protect the integrity of the SCORE program.

😺 the repliCATS organising team

You can contact Ans, Rania and Mel via: repliCATS-project@unimelb.edu.au

Preparation & reference material

For workshop week, you will assess claims and follow an elicitation protocol termed the ‘IDEA protocol’. The following materials will help you to understand the protocol, why we are using it, and what we will ask of you during the workshop.

Intro video & audio files (NB. this is from the June workshop. We will update the video on 20 July!)

 

The link to the platform is: https://replicats-bushel.eresearch.unimelb.edu.au/

The following documents are available via the Resources for participants page and may be useful if read before the introduction to the workshop. They can be easily accessed via the platform by clicking the resources button 🎓 :

  • repliCATS_BushelGuide_2021, plus a guide to round 1 and guide to round 2
  • Participant cheat sheet
  • Code of conduct

FAQs about the workshop

The following FAQs are specific to this virtual workshop. For general FAQs about answering claims, check the participant resources and the platform.

  • What if I don’t want to use Slack?
  • What can I discuss on Slack?
  • Who is in my group? And who is my facilitator?
  • How do I know when I have assessed all papers?
  • What if I become ill or cannot finish the workshop?
  • What if I can’t finish all papers on time?
  • What if I can’t attend my group’s discussion meeting?
  • I don’t like the claims assigned to my group.
  • I missed the introduction. Can I catch up?
  • Where or how can I give feedback?
  • What if I no longer want to take part?

What if I don’t want to use Slack?

We will be using slack to create a sense of fun and community as well as answering any questions you might have. However, if you do not want to use Slack you can send questions via email to repliCATS-project@unimelb.edu.au

You can also access the most important information you need to participate via the platform’s community page – there is a link to this web post. The community button (👥) is at the top right of your page once you have logged into the online platform. It is the 4th button along.

What can I discuss on Slack?

Discuss on Slack:

  • Questions about how to use the platform
  • Questions about the workshop generally
  • Clarifying the meaning of the questions

Save for the repliCATS platform & group discussion:

  • Descriptions of your reasons when evaluating a paper
  • Responses to other people’s justifications
  • Any information or discussion you might to have about a specific paper

How do I know when I have evaluated all the papers?

This should be really easy to track because each group of 4-6 people will be assigned 2-3 papers in total. We will ask you to do the following during the workshop:

  • Complete round 1 by 25 July
  • Complete discussion and round 2 by 27 July
  • We will contact you on Sunday with an update on your progress. We will contact you again after the assessment period has ended confirming that you have reached your goal.

You can also check the status of a paper at any time in the platform by hovering over the little harvey balls on the top right of each paper. It will tell you how many people have completed round 1. See question below.

How do I know how many people are already in the paper?

The easiest way to tell is to hover over the little pie chart  (or harvey ball) on the homepage of the platform. It looks like this.

It will tell you how many people are assessing the paper, and which round they are in. The first number will always be the number of people in your group, the second number is how many people have saved a round 1 response, the third number how many have submitted a round 1 response, the fourth number is how many people have submitted a round 2 response.

What if I become ill?

We’re sorry to hear that, and hope you have mild symptoms. Please let us know, and we’ll work out a way for you to keep your grant that doesn’t involve you assessing papers while you are unwell!

See: answer below.

What if I can’t finish the papers on time?

Don’t worry – we know that things don’t always go according to plan.

As much as possible, we’d like for everyone to assess the papers assigned to you in the week. If you think you’ll struggle to make it, please reach out to us on repliCATS-contact@unimelb.edu.au and we’ll work something out with you.

What if I can’t attend my group’s discussion meeting?

We’d really like you to try and make your group’s discussion. If you can’t attend ––that is a shame because participants have consistently told us that discussing papers is one of the most fun and enriching parts of our workshops. However, we understand that when you are all working virtually/from home lots of unanticipated things might come up.

If you can’t make the scheduled discussion time, that’s OK. Let your facilitator know as soon as possible. You can still update your round 2 after the end of the workshop — you will still have the benefit of reviewing your group’s responses and reasoning from round 1 before doing so.

I don’t like the papers assigned to my group.

We’ve tried our best to allocate papers that match the group’s expertise. We will not change the papers that have been assigned to your group except if you have been assigned a paper that you are working on a replication study for with TA1 or the Center for Open Science.

What if I no longer want to take part?

We’d be really sad to see you leave during assessment week, but we understand that things come up. Please let us know as soon as possible if you no longer want to take part, so we can offer the assessment grant to someone else on the waitlist.

I missed the introduction. Can I catch up?

Yes we will post the link to the recorded version in Slack and in the web post.

Where or how can I give feedback?

We’ll send you a link to an evaluation once the week is over. We’d love to know your thoughts and comments on the whole process.

Facilitator FAQs

Who’s in my group?

We have sent you an email with your group name and the link to the Facilitator Spreadsheet (also pinned to this post!). This contains all the information about your group and its members. If you can’t find the email, let us know via repliCATS-contact@unimelb.edu.au and we will resend you those details.

How do I contact my group members?

You will have a private group channel on Slack. If one or more of your group members does not want to use Slack, make sure you reach out to them via email to introduce yourself. They might also contact you via email.

I don’t know the answer to a question from one of my group members!

The #facilitator channel on Slack will be the hive mind of facilitators. Before you contact the repliCATS team, check the channel for info. It is likely that your question will have been posted before and answered. If not, ask us at repliCATS-contact@unimelb.edu.au.

When should I schedule my group’s discussion session?
People are busy, so try to settle on a date and time as soon as you can. We recommend scheduling the discussion for Monday 26 July or Tuesday 27 July to give people ample time to complete their Round 1 judgements.

What if someone can’t attend the discussion?

We encourage all group members to attend the discussion, but it might not be possible to find a date and time that suits everyone, and sometimes people’s plans change at the last minute (e.g. they might have responsibilities as a carer). It is helpful for us to know how many people attended the discussion, so please log attendance on the Facilitator Spreadsheet (pinned to the post in the slack channel).

I have never facilitated a discussion before – what do I do?

Your main role is to ensure that everyone gets a chance to talk and to keep the discussion focused on exploring the reasoning that underpins Round 1 judgements. For more detailed guidance, review the facilitator training (a recording can be found here) and the Guidelines for Facilitators (pinned to the post in the slack channel)

When does Round 2 start?

Whenever your group has discussed a paper, participants can start updating their estimates. Whether you want to do this in real-time, i.e. as you are discussing the questions, or after the discussion is finished, is up to you.

What happens at the end of the workshop?

Please be sure to thank your group on behalf of the repliCATS team. Stay tuned for a wrap-up event. It will be different and fun, but as usual, there will be awards! Remind your participants that it might be up to 6 weeks for their assessment grants to be processed.


Welcome to new workshop coordinators!

We’re really excited to have Ans Vercammen & Rania Poulis join the repliCATS project team to help coordinate our 2021 bushel workshops.

Ans VercammenAns Vercammen is trained as a research psychologist and completed a PhD in behavioural and cognitive neuroscience, studying the origins of auditory hallucinations. More recently, her interests have shifted away from neuropsychology to understanding why people (don’t) make pro-environmental choices and how to support environmental decision making at various scales. After completing an MSc in Conservation Science, she joined the Centre for Environmental Policy at Imperial College London, working with Mark Burgman on expert judgement. She recently joined the repliCATS Project to help coordinate Phase 2 workshops and conduct qualitative analyses as part of the Reasoning Team.

 

Rania PoulisRania Poulis has had a varied career beginning as a research assistant with a molecular biology/genomics background. She spent many years working to increase and improve gene-specific databases; this included a large amount of international conference organisation for the purpose of improving healthcare under the auspices of the Human Variome Project, an NGO of UNESCO. More recently Rania has been organising conferences for various scientific organisations and joined the team to help coordinate Phase 2 workshops and organise the AIMOS conference.


pre-SIPS repliCATS workshop – June 15-22

Hi folks, this post has all the info you need if you are participating in the pre-SIPS repliCATS bushel virtual workshop running from 15-22 June!

Here are links to all the information you’ll lead leading up to and during the week:

About the workshop

The workshop will run from 15-22 June (for your timezone), and during that time you’ll be asked to assess 2-3 bushel papers from psychology and adjacent disciplines. You’ll work in small groups of 4-6 people, each with a dedicated facilitator.

Here’s how it will work:

  • Before kick-off – you’ll get invited to the workshop slack space, receive your repliCATS platform sign-up details, be introduced to your group members & facilitator, and confirm your banking details so we can pay you the US$200 grants! (30-40 mins)
  • 15 June – workshop kick-off & introduction session* (45 mins) See below for details.
  • 15-20 June – complete your round 1 judgements (~2-2.5 hours, we don’t expect you to work on the weekends, but you have that extra time in case…)
  • 21-22 June – group discussion and submit your round 2 judgements (~1-1.5 hours). Group facilitators will contact you to schedule a synchronous virtual discussion at a time that suits your group on either Monday 21 June or Tuesday 22 June.
  • We’ll have a dedicated Slack space for everyone — there will be some friendly comps, some #random replipets fun.
  • Wrap-up & your feedback. We have something fun and new planned 🙂

*There will be two live sessions accommodating different time zones – you only need to turn up to one 🙂

Things to do before 07 June

  • Let us know if your availability changes at any time.
  • Complete consent & demographics. Before the workshop you must set up your account on the repliCATS platform, which includes completing a short demographics component and a quiz. This will take about 20-30 minutes, and must be completed in one sitting. The e-mail address you use on this form will be your repliCATS platform username too. At the end of the survey select “complete” from the options available to you.
  • Your banking details so we can pay you.
    • New participants: send us your first and last name as they appear on your bank account, and look for an e-mail from the University to set you up as a supplier.
    • Returning participants: you don’t need to do anything unless your bank details have changed.
  • Save the dates for the workshop introductions. These introductions will take you through everything you need to know! To accommodate different time zones, we are running two (identical) live webinars to kick-off the workshop. We will also record them for you to watch later.
Melbourne New Delhi Singapore London Washington DC Los Angeles Zoom links
Intro #1 Tue 15 Jun, 8am Tue 15 Jun, 3am Tue 15 Jun, 6am Mon 14 Jun, 11pm Mon 14 Jun, 6pm Mon 14 Jun, 3pm Check e-mail & slack
Intro #2 (repeat) Tue 15 Jun, 6pm Tue 15 Jun, 1.30pm Tue 15 Jun, 4pm Tue 15 Jun, 9am Tue 15 Jun, 4am Tue 15 Jun, 1am Check e-mail & slack

Find for the exact time for your city: https://www.timeanddate.com/worldclock/fixedform.html

  • Accept invitation into the slack workspace. For the week, we will have a dedicated slack workspace. Please try to use the same email address as the one you used for your consent & demographics so we know it’s you! Don’t want to use slack? See our faqs below.

Communication

👩🏻‍💻 repliCATS on call: Live Q&A troubleshooting assistance (AEST) on slack

We will have repliCATS staff members available during working hours throughout the week.

The slack channel is for communicating and troubleshooting (and fun) during the week, and is separate to the reasoning, discussion, and comments we ask you to provide while assessing claims on our repliCATS platform.

When you evaluate claims within the online platform you will have an avatar that protects your anonymity and the data are recorded for our project. On Slack, you are identifiable so you can get to know each other. We will not be recording your Slack conversations. We want to understand the way you think and reason about claims so we would really appreciate it if you keep any information about this on the repliCATS platform and for your discussion sessions.

If you have trouble with your Slack channel please contact us, repliCATS-project@unimelb.edu.au

#️⃣ twitter

You can tweet using #repliCATS2021 @repliCATS

Please don’t share estimates or claim information publicly, this helps us protect the integrity of the SCORE program.

😺 the repliCATS organising team

You can contact Ans, Rania and Mel via: repliCATS-project@unimelb.edu.au

Preparation & reference material

For workshop week, you will assess claims and follow an elicitation protocol termed the ‘IDEA protocol’. The following materials will help you to understand the protocol, why we are using it, and what we will ask of you during the workshop.

Intro video & audio files

 

The link to the platform is: https://replicats-bushel.eresearch.unimelb.edu.au/

The following documents are available via the Resources for participants page and may be useful if read before the introduction to the workshop. They can be easily accessed via the platform by clicking the resources button 🎓 :

  • repliCATS_BushelGuide_2021, plus a guide to round 1 and guide to round 2
  • Participant cheat sheet
  • Code of conduct

FAQs about the workshop

The following FAQs are specific to this virtual workshop. For general FAQs about answering claims, check the participant resources and the platform.

  • What if I don’t want to use Slack?
  • What can I discuss on Slack?
  • Who is in my group? And who is my facilitator?
  • How do I know when I have assessed all papers?
  • What if I become ill or cannot finish the workshop?
  • What if I can’t finish all papers on time?
  • What if I can’t attend my group’s discussion meeting?
  • I don’t like the claims assigned to my group.
  • I missed the introduction. Can I catch up?
  • Where or how can I give feedback?
  • What if I no longer want to take part?

What if I don’t want to use Slack?

We will be using slack to create a sense of fun and community as well as answering any questions you might have. However, if you do not want to use Slack you can send questions via email to repliCATS-project@unimelb.edu.au

You can also access the most important information you need to participate via the platform’s community page – there is a link to this web post. The community button (👥) is at the top right of your page once you have logged into the online platform. It is the 4th button along.

What can I discuss on Slack?

Discuss on Slack:

  • Questions about how to use the platform
  • Questions about the workshop generally
  • Clarifying the meaning of the questions

Save for the repliCATS platform & group discussion:

  • Descriptions of your reasons when evaluating a paper
  • Responses to other people’s justifications
  • Any information or discussion you might to have about a specific paper

How do I know when I have evaluated all the papers?

This should be really easy to track because each group of 4-6 people will be assigned 2-3 papers in total. We will ask you to do the following during the workshop:

Complete round 1 by 20 June
Complete discussion and round 2 by 22 June
We will contact you on Sunday, 20 June with an update on your progress. We will contact you again after the assessment period has ended confirming that you have reached your goal.

You can also check the status of a paper at any time in the platform by hovering over the little harvey balls on the top right of each paper. It will tell you how many people have completed round 1. See question below.

How do I know how many people are already in the paper?

The easiest way to tell is to hover over the little pie chart  (or harvey ball) on the homepage of the platform. It looks like this.

It will tell you how many people are assessing the paper, and which round they are in. The first number will always be the number of people in your group, the second number is how many people have saved a round 1 response, the third number how many have submitted a round 1 response, the fourth number is how many people have submitted a round 2 response.

What if I become ill?

We’re sorry to hear that, and hope you have mild symptoms. Please let us know, and we’ll work out a way for you to keep your grant that doesn’t involve you assessing papers while you are unwell!

See: answer below.

What if I can’t finish the papers on time?

Don’t worry – we know that things don’t always go according to plan.

As much as possible, we’d like for everyone to assess the papers assigned to you in the week. If you think you’ll struggle to make it, please reach out to us on repliCATS-contact@unimelb.edu.au and we’ll work something out with you.

What if I can’t attend my group’s discussion meeting?

We’d really like you to try and make your group’s discussion. If you can’t attend ––that is a shame because participants have consistently told us that discussing papers is one of the most fun and enriching parts of our workshops. However, we understand that when you are all working virtually/from home lots of unanticipated things might come up.

If you can’t make the scheduled discussion time, that’s OK. Let your facilitator know as soon as possible. You can still update your round 2 after 20 June — you will still have the benefit of reviewing your group’s responses and reasoning from round 1 before doing so.

I don’t like the papers assigned to my group.

We’ve tried our best to allocate papers that match the group’s expertise. We will not change the papers that have been assigned to your group except if you have been assigned a paper that you are working on a replication study for with TA1 or the Center for Open Science.

What if I no longer want to take part?

We’d be really sad to see you leave during assessment week, but we understand that things come up. Please let us know as soon as possible if you no longer want to take part, so we can offer the assessment grant to someone else on the waitlist.

I missed the introduction. Can I catch up?

Yes we will post the link to the recorded version in Slack and in the web post.

Where or how can I give feedback?

We’ll send you a link to an evaluation once the week is over. We’d love to know your thoughts and comments on the whole process.

Facilitator FAQs

Who’s in my group?

We have sent you an email with your group name and the link to the Facilitator Spreadsheet (also pinned to this post!). This contains all the information about your group and its members. If you can’t find the email, let us know via repliCATS-contact@unimelb.edu.au and we will resend you those details.

How do I contact my group members?

You will have a private group channel on Slack. If one or more of your group members does not want to use Slack, make sure you reach out to them via email to introduce yourself. They might also contact you via email.

I don’t know the answer to a question from one of my group members!

The #facilitator channel on Slack will be the hive mind of facilitators. Before you contact the repliCATS team, check the channel for info. It is likely that your question will have been posted before and answered. If not, ask us at repliCATS-contact@unimelb.edu.au.

When should I schedule my group’s discussion session?
People are busy, so try to settle on a date and time as soon as you can. We recommend scheduling the discussion for Monday 21 June or Tuesday 22 June to give people ample time to complete their Round 1 judgements.

What if someone can’t attend the discussion?

We encourage all group members to attend the discussion, but it might not be possible to find a date and time that suits everyone, and sometimes people’s plans change at the last minute (e.g. they might have responsibilities as a carer). It is helpful for us to know how many people attended the discussion, so please log attendance on the Facilitator Spreadsheet (pinned to the post in the slack channel).

I have never facilitated a discussion before – what do I do?

Your main role is to ensure that everyone gets a chance to talk and to keep the discussion focused on exploring the reasoning that underpins Round 1 judgements. For more detailed guidance, review the facilitator training (a recording can be found here) and the Guidelines for Facilitators (pinned to the post in the slack channel)

When does Round 2 start?

Whenever your group has discussed a paper, participants can start updating their estimates. Whether you want to do this in real-time, i.e. as you are discussing the questions, or after the discussion is finished, is up to you.

What happens at the end of the workshop?

Please be sure to thank your group on behalf of the repliCATS team. Stay tuned for a wrap-up event. It will be different and fun, but as usual, there will be awards! Remind your participants that it might be up to 6 weeks for their assessment grants to be processed.


A SCORE of pre-prints.

In phase 1, we crowdsourced judgements of replicability for 3000 research claims drawn from peer-reviewed journals across eight social and behavioural science disciplines. The following thread unrolls a series of pre-prints now available describing the overall SCORE collaborative in phase 1 (involving five research teams from around the world), as well as a number of articles from the repliCATS project team, including:


First 2021 workshop announced!

repliCATS workshops will be back in June! We are kicking off our phase 2 research with a pre-SIPS repliCATS virtual workshop from 15-22 June 2021.

What’s new?
In 2021, the repliCATS workshops will focus on evaluating the credibility of entire published research papers (or what we are calling “bushel papers”) in the social and behavioural sciences literature. For the pre-SIPS workshop, the papers you evaluate will be drawn from psychology and psychology-adjacent fields.

However, our workshops will remain familiar to returning participants. Working in small groups of 4-6, you will be asked to evaluate 2-3 papers. Each group will have its own facilitator, who will guide you through a structured elicitation protocol.

For more details and to express interest in participating in this workshop, use the following form:

 



SCORE program renewed for phase 2

We’re excited to announce that DARPA has renewed the SCORE program—and the repliCATS project—for a second phase!

What does that mean? It means your predictions met accuracy thresholds! And we’ll return in 2021. In phase 2, we’ll evaluate a fresh set of claims for likely replicability as well as ask new questions, for example, about validity and generalisability. We’ll continue our work developing the repliCATS protocol for alternative rapid peer review, and peer review training using the opportunity for feedback & calibration that our protocol provides.
All of our achievements to date are attributable to the 550+ participants who have collaborated with us so far. We hope you’ll continue to join our workshops, and earn our badges, and keep us accountable.

For more information about the wider SCORE program, see: https://www.darpa.mil/program/systematizing-confidence-in-open-research-and-evidence


Number of posts found: 34