Closed: Express interest in assessing COVID-19 research claims

The repliCATS project — and the wider SCORE program — has been expanded by DARPA to include 100 COVID-19 research claims from the social and behavioural sciences.

Governments around the world may rely on social and behavioural science research to understand crises and inform policies. Inclusion of COVID claims will ensure that the repliCATS project is tested on a range of social and behavioural science or SBS research areas, making it uniquely tailored for the critical evaluation of a broad spectrum of research claims.

The repliCATS project will run a virtual worksop from 2-8 September to assess these COVID-19 claims, and are looking for 100 participants.

Are you new to the repliCATS project? If so, you can watch our chief investigator, Prof Fiona Fidler introduce the project, or read more about what it means to assess research claims.

Express interest in workshop (closed) by Friday, 14 August

Expressions of interest in participating in our workshop are now open, but places are capped at 100 participants.

**UPDATE: EOIs closed on 14 August. Thank you so much for expressing interest. You will have received an e-mail letting you know if you have been confirmed for the workshop or if you are on a waitlist. Rolling offers were made on a first-come basis. If you have any questions, please contact us on repliCATS-project@unimelb.edu.au**

The key details again (in case you missed them).

  • There are 100 x USD200 (AUD 280) assessment grants available for eligible participants.
  • You need to be available between 2-8 September, for ~ 6 hours over the week.
  • *Each participant will be asked to assess ten claims.
  • You can assess claims when it suits you.
  • You’ll work in groups of 3-6 other participants, using our online platform.
  • If this is your first time assessing claims, don’t worry. We’ll run an intro session to help you get started, and be available to help you over the week.
  • *If you have already expressed interest before 03 August, you do not need to re-apply. We will contact you shortly with additional details about the workshop, including assessment grant eligibility.

Form not working? Click here.


National Science Week: Assessing the reliability of COVID research – a repliCATS webinar

As part of National Science Week, join the repliCATS project team for a peak into how and why we’re assessing COVID-19 claims.

Responding to crises like the COVID-19 pandemic requires a strong evidence basis. But which scientific results are trustworthy? During this crisis there have been high profile cases of scientific misconduct and retractions and more general concerns about the quality of research being generated quickly. This is set against a backdrop of the ‘replication crisis’ which has shown low rates of replicability for research claims in several scientific disciplines.

Details:

Date: Thursday 20 August 2020
Time: 12:00pm – 1:30pm
Location: Webinar
Cost: Free

NB: this webinar is separate to the virtual workshop that we will run between 2-8 September.

For more information, and to register: https://events.unimelb.edu.au/science/event/7318-assessing-the-reliability-of-covid-research-a


Reimagining trust in science panel (19 Aug), feat Fiona Fidler

Recent crises have prompted discussion on ‘Trust in Science’. This raises the question: is science the kind of thing that can be trusted?

Is ‘Trust in Science’ the same thing as trusting individual scientists or research organisations, or is it something else? How do we maintain ‘Trust in Science’ while holding a reasonable level of uncertainty about individual research results?

An understanding of ‘Trust in Science’ needs to incorporate the emotional and political engagements revealed y public attitudes, the contested nature of professional identity and authority in contemporary society, and recent work on the ‘replication crisis’. This History and Philosophy of Science seminar will discuss recent work on the philosophy of trust in science.

Speakers will include:

For more information, visit:

https://events.unimelb.edu.au/science/event/7315-reimagining-trust-in-science-a-history-and


Monthly winners!

FINAL round winners: 1 – 30 July

This was our final repliCATS remote round for phase 1. And at the end, we reached our goal of assessing 3000 claims! It was a phenomenal effort, with 1840 assessments made in 571 claims.

This round returned to basics, awarding the three participants who assessed the most number of claims.

Congratulations to:

  • sugarglider_419, who assessed an incredible 211 claims(!) to win US$500

as well as

  • crocodile_280, who assessed 102 claims, and
  • echidna_219, who assessed 55 claims, to win US$250 each.

Round 6 winners: 1 – 30 June

This round was all about business and economics claims! In total, participants submitted 362 assessments in eligible claims. (And another 798 assessments in claims from other disciplines.)

Congratulations to:

  • sugarglider_419, who assessed a whopping 77 business and economics claims and wins US$500

as well as

  • cockatoo_202, who assessed 41, and
  • dingo_268, who assessed 28 claims, to win US$250 each.

Round 5 winners: 22 April – 22 May

We intended to run a lottery for everyone who completed 10 or more claims outside our virtual workshop. However, with so many of our wonderful participants concentrating on our virtual workshop workshop, no one was eligible for this month’s prize.

Round 4 winners: 23 Mar – 22 Apr 2020

We ran a lottery, and had 17 participants who completed 10+ assessments over this period. However, for several reasons only two out of those 17 participants were eligible to win a prize.

A number of our participants fall under an ethics agreement which doesn’t allow them to win these prizes, so they continue to win kudos and gratitude and this emoji: 😻!

Now to the winners — it was a possum double act! Congrats to:

  • possum_396 and
  • possum_745

who both win US$250 for assessing 10+ claims in round 3!

Round 3 was a quieter round (which is to be expected given we had to reschedule so many planned workshops). We only had 39 participants who contributed assessments, but they made assessments across 601 claims!

Round 5 (23 Apr – 22 May) is another lottery! Click here for more information on this month’s prizes!

Round 3 winners: 18 Feb – 22 Mar 2020

We ran a lottery, and had 23 participants who completed 10+ assessments over this period. However, for several reasons only two out of those 23 participants were eligible to win a prize.

A number of our participants fall under an ethics agreement which doesn’t allow them to win these prizes, so instead they win kudos and gratitude and this emoji: 😻!

Now to the winners. Congrats to:

  • possum_396 and
  • goanna_685

who both win US$250 for assessing 10+ claims in round 3!

In total for round 3, we had 140 participants who contributed assessments across 471 claims!

Round 2 winners: 17 Jan – 18 Feb 2020

Congrats to:

  • numbat_339 for submitting 21 unique reasons to support their judgements over the round, who wins US$500!
  • possum_396 for the second most number — 15 — of reasons submitted (US$250)
  • and, thornydevil_715 gets third place with 12 (US$250).

Round 1 winners: 9 Dec – 17 Jan 2020

Our first round fo remote claims assessment ended on 17 Jan 2020.

A huge congrats to:

  • goanna_685 for assessing a whopping 30 claims, and winning US$800!
  • pobblebonk_711 for assessing 19 claims (US$250)
  • and a tie-break gave pobblebonk_699 third place (US$250).

Round 3 is a lottery! And closes on 17 March 2020. Click here for more information on this month’s prizes!

 



Watch 📺 Fiona Fidler gives RIOT Science talk on all things repliCATS

Earlier this month, Fiona Fidler gave a talk about the repliCATS project, and some of our plans for the future of the project. If you missed it (including if you were asleep because it ran from 11am-12pm BST), you can re-watch it via this link (YouTube)!

Talk abstract

The repliCATS project evaluates published scientific research. As the acronym—Collaborative Assessments for Trustworthy Science—suggests, repliCATS is a group activity, centred around assessing the trustworthiness of research claims. Reviewers first make private individual assessments about a research claim—judging its comprehensibility, the prior plausibility of underlying effect, and its likely replicability. Reviewers then share their judgements and reasoning with group members, providing both new information and the opportunity for feedback and calibration. The group interrogates differences in opinion and explores counterfactuals. After discussion, there is a final opportunity for privately updating individual judgements. Importantly the repliCATS process is not consensus-driven – reviewers can disagree, and their ratings and probability judgements are mathematically aggregated into a final assessment. At the moment, the repliCATS platform exists primarily to predict replicability. Launched in January 2019 as part of the DARPA SCORE program, over 18 months repliCATS elicited group assessments and captured associated reasoning and discussion, for 3,000 published social scientific research claims in 8 disciplines (Business, Criminology, Economics, Education, Political Science, Psychology, Public Administration, and Sociology). The repliCATS team are now working to extend the platform beyond merely predicting replicability, to deliver a more comprehensive peer review protocol. Suspected advantages of a repliCATS process over traditional peer review include: inbuilt training and calibration; feedback that is intrinsically rewarding; an inherently interactive process, but one which does not implicitly rely on ‘consensus by fatigue’; and a process that actively encourages interrogation. This talk will present some preliminary findings, and discuss the future of the platform.


💬 participant tweets we love


repliCATS monthly prizes and rules

Round 7 (final round!): 1 July – 30 July 2020

It is the final remote round for repliCATS in phase 1! So, we’ll end how we started, by giving prizes to the three participants who assess the most number of total claims in July 2020, as follows:

US$500 – for the participant who assessed the most number of claims
US$250 – for the next two highest participants.

Round 6: 1 June – 30 June 2020 – assess the most business or economics claims!

In this round we’ll calculate how many business or economics** claims people complete between 1 June – 30 June 2020 (midnight, AEST)

  • US$500 – for the person who completes the most*
  • US$250 – for the people who completed 2nd and 3rd most business or economics claims *

*A completed claim counts as submitting a round 2 assessment.

** Claims are considered Business (marketing) or Economics claims if they were from the journals specified the table in FAQs under “From which journals are the 3000 claims being chosen?”

Follow these links for:

*NB: each month the prizes might change.

How it works (V1 – 9 Dec 2019 V2 – 05 April 2020)

  • Prizes are awarded monthly, and the terms of the prize will be announced monthly when the prizes for the previous month are released
  • Prizes are based on participation in that month
  • Prizes are awarded to the top three participants based on monthly reward
  • There will always be a winner. Tie-break rules: in case of ties, participation metrics will be applied to the tie-break, e.g. number of claims, number of unique comments, number of up-votes
  • If tie-break rules are applied, the metrics used to determine winners will be announced
  • Winners will only be named by their platform avatar name (i.e. they will remain anonymous)
  • A list of winners each month will be published on repliCATS website [link to website], and posted to the platform community page
  • The first time you win, you will be awarded a “cat-as-trophy” badge which will appear on your profile in the community page

Eligibility

  • User has not violated the repliCATS_CodeofConduct
  • All repliCATS workshop participants are eligible for the prizes in the month of the workshop
  • No repliCATS paid staff members are eligible
  • Participants can win multiple times, in multiple categories (if applicable)

Spirit of participation

  • Your judgements and reasoning will contribute to the data being collected for the each of the 3,000 claims being assessed for the repliCATS project
  • We trust you won’t cheat i.e. each user will only have one account and no-one will share accounts
  • You will abide by the repliCATS_CodeofConduct

Payment

  • The repliCATS project will process winners’ payments in the next monthly financial cycle, after receiving necessary documentation from winner
    • If winners do not provide the necessary documentation within 2 months of notification or the repliCATS project phase completing (whichever comes first), they are no longer eligible for the prize (added 05 April 2020)
  • The repliCATS project will not incur any recipient’s bank fees
  • Payments will be subject to applicable exchange rate, payments are made in USD

Past months categories

December 2019 – US$1000 in prizes for assessing the most number of claims

In our first month of launching the remote group platform, we’ll give prizes to the three participants who assess the most number of total claims in the period 9 Dec 2019 – 9 Jan 2020, as follows:

  • US$800 – for the participant who assessed the most number of claims
  • US$250 – for the next two highest participants.

January 2020 – US$1000 for the number of unique reasons provided

In this round from 17 Jan – 17 Feb 2020 we’ll give prizes to the three participants who provide the most number of unique reasons as follows:

  • US$500 – for the participant who assessed the most number of claims*
  • US$250 – for the next two highest participants.*

February 2020 – complete 10 claims, US$250 lottery draw for four participants

In this round from 18 Feb – 17 22 Mar 2020 (midnight, AEST) we’ll put all participants who complete at least 10 claims into a pool, and randomly draw four winners from that pool.

  • US$250 – for each person drawn this month*
    • A completed claim counts as submitting a round 2 assessment.

March 2020 – complete 10 or more claims, US$250 lottery for two four participants

In this round from 23 Mar 2020 – 22 April (midnight, AEST) we’ll put all participants who complete at least 10 claims into a pool, and randomly draw four winners from that pool.

  • US$250 – for each person drawn this month*
    • A completed claim counts as submitting a round 2 assessment.

Round 5: 23 April – 22 May 2020 – complete 10 or more claims, US$250 lottery for up to four participants

In this round from 23 Apr 2020 – 22 May (midnight, AEST) we’ll put all participants who complete at least 10 claims into a pool, and randomly draw up to four winners from that pool.

  • US$250 – for each person drawn this month*
    • A completed claim counts as submitting a round 2 assessment.

repliCATS workshop went virtual – over 550 claims assessed in the week

Our last phase 1 workshop was originally scheduled to run before the 2020 Society for Improving Psychological Sciences (SIPS) conference in Victoria, Canada. Then everything changed.

The repliCATS team created our first virtual workshop designed to accomodate all the original workshop participants scattered around over 10 timezones. What we ended up with was repliCATS2020 workshop week that ran from 12-19 May, where over 150 participants and facilitators contributed 2238 individual judgements, or the equivalent of 560 claims!

That was more than what we completed at the last SIPS face-to-face workshop in 2019.

Of course, that didn’t stop participants re-creating their favourite bits of our face-to-face workshops.

The repliCATS team had a few surprises for participants though: we reprised our very popular closing ceremony and awards virtually, including a musical finale that we cannot reproduce here due to copyright issues :D. You’ll just have to take Julia and Dax’s words for it.

The repliCATS team have learned a lot from running this first virtual workshop, which required some significant adjustments compared to face-to-face equivalents. However, participant feedback has left us feeling that we responded successfully, and have a platform upon which we can run future workshops virtually without compromising on participant engagement. Of course some of this success is thanks to our participants themselves who engaged in this new format enthusiastically and with an open mind. So, all that is left to report is a big thank you to the SIPS community. We’re humbled by the contribution this community has made across two huge face-to-face workshops in 2019 and 2020, contributing to 1100 of the 3000 claims we’re aiming to assess!

What’s next for repliCATS?

While this might have been the last big workshop the repliCATS team will run in phase 1 of the SCORE program, we have a little bit left to go. As on 22 June, we have assessments in over 2775 of the 3000 claims. We are hoping to complete assessments by early August 2020, and the repliCATS remote platform will remain live and operational for a bit longer.


Platform outage on 13 May. If you assessed claims between 12-13 May, please read this.

Hi repliCATS participants, some routine system maintenance being undertaken on the repliCATS platform in the afternoon on 13 May (Australian time) resulted in our database being wiped from the live platform.

We have identified the problem, and restored the platform shortly after, but we have irretrievably lost all of the assessments you submitted on the platform on 12 May and the first part of 13 May (Australian time). Your private information and demographic information is stored separately and has not been compromised in any way, but we’ve lost all any assessments submitted for a 24-36 hour period.

We’re so sorry because we appreciate the work that goes into each assessment you make on the platform. We have never experienced a technical issue like this before and have no reason to expect a repeat occurrence.

You can re-do/re-submit assessments for the claims that you already assessed, if only if to re-enter the quantitative judgements for these lost claims.

If you have any questions or would like to discuss this further, please contact us: repliCATS-contact@unimelb.edu.au and we will get back to you as soon as possible.


Number of posts found: 32