In phase 1, we crowdsourced judgements of replicability for 3000 research claims drawn from peer-reviewed journals across eight social and behavioural science disciplines. The following thread unrolls a series of pre-prints now available describing the overall SCORE collaborative in phase 1 (involving five research teams from around the world), as well as a number of articles from the repliCATS project team, including:
repliCATS workshops will be back in June! We are kicking off our phase 2 research with a pre-SIPS repliCATS virtual workshop from 15-22 June 2021.
What's new?
In 2021, the repliCATS workshops will focus on evaluating the credibility of entire published research papers (or what we are calling "bushel papers") in the social and behavioural sciences literature. For the pre-SIPS workshop, the papers you evaluate will be drawn from psychology and psychology-adjacent fields.
However, our workshops will remain familiar to returning participants. Working in small groups of 4-6, you will be asked to evaluate 2-3 papers. Each group will have its own facilitator, who will guide you through a structured elicitation protocol.
For more details and to express interest in participating in this workshop, use the following form:
We’re excited to announce that DARPA has renewed the SCORE program—and the repliCATS project—for a second phase!
What does that mean? It means your predictions met accuracy thresholds! And we’ll return in 2021. In phase 2, we’ll evaluate a fresh set of claims for likely replicability as well as ask new questions, for example, about validity and generalisability. We’ll continue our work developing the repliCATS protocol for alternative rapid peer review, and peer review training using the opportunity for feedback & calibration that our protocol provides.
All of our achievements to date are attributable to the 550+ participants who have collaborated with us so far. We hope you’ll continue to join our workshops, and earn our badges, and keep us accountable.
For more information about the wider SCORE program, see: https://www.darpa.mil/program/systematizing-confidence-in-open-research-and-evidence
Join our CI Fiona Fidler & philosopher of science Rachel Brown for a webinar on "Metascience should not be defined by its methods.
Metascience, or metaresearch, is a field of research that has grown out of the replication crisis. Amongst other things, metascience evaluates and monitors open science initiatives and other interventions to improve scientific practices and cultures. It studies the impact of institutional reward structures, performance metrics, incentives, and methods of scientific resource allocation. It also evaluates peer review practices—both current, and alternative future proposals. As a field, it is (unfortunately) often defined as “the scientific study of science itself” or “the science of doing science”. In this talk I’ll explain why metascience shouldn’t be defined by appeal to a monolithic scientific method, and discuss other ways of conceptualizing what this new epistemic community does.
The webinar is now available to watch here: https://biology.anu.edu.au/news-events/events/ee-webinar-metascience-should-not-be-defined-its-methods
The repliCATS project — and the wider SCORE program — has been expanded by DARPA to include 100 COVID-19 research claims from the social and behavioural sciences.
Governments around the world may rely on social and behavioural science research to understand crises and inform policies. Inclusion of COVID claims will ensure that the repliCATS project is tested on a range of social and behavioural science or SBS research areas, making it uniquely tailored for the critical evaluation of a broad spectrum of research claims.
The repliCATS project will run a virtual worksop from 2-8 September to assess these COVID-19 claims, and are looking for 100 participants.
Are you new to the repliCATS project? If so, you can watch our chief investigator, Prof Fiona Fidler introduce the project, or read more about what it means to assess research claims.
Express interest in workshop (closed) by Friday, 14 August
Expressions of interest in participating in our workshop are now open, but places are capped at 100 participants.
**UPDATE: EOIs closed on 14 August. Thank you so much for expressing interest. You will have received an e-mail letting you know if you have been confirmed for the workshop or if you are on a waitlist. Rolling offers were made on a first-come basis. If you have any questions, please contact us on repliCATS-project@unimelb.edu.au**
The key details again (in case you missed them).
- There are 100 x USD200 (AUD 280) assessment grants available for eligible participants.
- You need to be available between 2-8 September, for ~ 6 hours over the week.
- *Each participant will be asked to assess ten claims.
- You can assess claims when it suits you.
- You'll work in groups of 3-6 other participants, using our online platform.
- If this is your first time assessing claims, don't worry. We'll run an intro session to help you get started, and be available to help you over the week.
- *If you have already expressed interest before 03 August, you do not need to re-apply. We will contact you shortly with additional details about the workshop, including assessment grant eligibility.
Form not working? Click here.
As part of National Science Week, join the repliCATS project team for a peak into how and why we're assessing COVID-19 claims.
Responding to crises like the COVID-19 pandemic requires a strong evidence basis. But which scientific results are trustworthy? During this crisis there have been high profile cases of scientific misconduct and retractions and more general concerns about the quality of research being generated quickly. This is set against a backdrop of the ‘replication crisis’ which has shown low rates of replicability for research claims in several scientific disciplines.
Details:
Date: Thursday 20 August 2020
Time: 12:00pm - 1:30pm
Location: Webinar
Cost: Free
NB: this webinar is separate to the virtual workshop that we will run between 2-8 September.
For more information, and to register: https://events.unimelb.edu.au/science/event/7318-assessing-the-reliability-of-covid-research-a
Recent crises have prompted discussion on ‘Trust in Science’. This raises the question: is science the kind of thing that can be trusted?
Is ‘Trust in Science’ the same thing as trusting individual scientists or research organisations, or is it something else? How do we maintain ‘Trust in Science’ while holding a reasonable level of uncertainty about individual research results?
An understanding of ‘Trust in Science’ needs to incorporate the emotional and political engagements revealed y public attitudes, the contested nature of professional identity and authority in contemporary society, and recent work on the ‘replication crisis’. This History and Philosophy of Science seminar will discuss recent work on the philosophy of trust in science.
Speakers will include:
For more information, visit:
https://events.unimelb.edu.au/science/event/7315-reimagining-trust-in-science-a-history-and
FINAL round winners: 1 - 30 July
This was our final repliCATS remote round for phase 1. And at the end, we reached our goal of assessing 3000 claims! It was a phenomenal effort, with 1840 assessments made in 571 claims.
This round returned to basics, awarding the three participants who assessed the most number of claims.
Congratulations to:
- sugarglider_419, who assessed an incredible 211 claims(!) to win US$500
as well as
- crocodile_280, who assessed 102 claims, and
- echidna_219, who assessed 55 claims, to win US$250 each.
Round 6 winners: 1 - 30 June
This round was all about business and economics claims! In total, participants submitted 362 assessments in eligible claims. (And another 798 assessments in claims from other disciplines.)
Congratulations to:
- sugarglider_419, who assessed a whopping 77 business and economics claims and wins US$500
as well as
- cockatoo_202, who assessed 41, and
- dingo_268, who assessed 28 claims, to win US$250 each.
Round 5 winners: 22 April - 22 May
We intended to run a lottery for everyone who completed 10 or more claims outside our virtual workshop. However, with so many of our wonderful participants concentrating on our virtual workshop workshop, no one was eligible for this month's prize.
Round 4 winners: 23 Mar - 22 Apr 2020
We ran a lottery, and had 17 participants who completed 10+ assessments over this period. However, for several reasons only two out of those 17 participants were eligible to win a prize.
A number of our participants fall under an ethics agreement which doesn't allow them to win these prizes, so they continue to win kudos and gratitude and this emoji: 😻!
Now to the winners -- it was a possum double act! Congrats to:
- possum_396 and
- possum_745
who both win US$250 for assessing 10+ claims in round 3!
Round 3 was a quieter round (which is to be expected given we had to reschedule so many planned workshops). We only had 39 participants who contributed assessments, but they made assessments across 601 claims!
Round 5 (23 Apr - 22 May) is another lottery! Click here for more information on this month's prizes!
Round 3 winners: 18 Feb - 22 Mar 2020
We ran a lottery, and had 23 participants who completed 10+ assessments over this period. However, for several reasons only two out of those 23 participants were eligible to win a prize.
A number of our participants fall under an ethics agreement which doesn't allow them to win these prizes, so instead they win kudos and gratitude and this emoji: 😻!
Now to the winners. Congrats to:
- possum_396 and
- goanna_685
who both win US$250 for assessing 10+ claims in round 3!
In total for round 3, we had 140 participants who contributed assessments across 471 claims!
Round 2 winners: 17 Jan - 18 Feb 2020
Congrats to:
- numbat_339 for submitting 21 unique reasons to support their judgements over the round, who wins US$500!
- possum_396 for the second most number -- 15 -- of reasons submitted (US$250)
- and, thornydevil_715 gets third place with 12 (US$250).
Round 1 winners: 9 Dec - 17 Jan 2020
Our first round fo remote claims assessment ended on 17 Jan 2020.
A huge congrats to:
- goanna_685 for assessing a whopping 30 claims, and winning US$800!
- pobblebonk_711 for assessing 19 claims (US$250)
- and a tie-break gave pobblebonk_699 third place (US$250).
Round 3 is a lottery! And closes on 17 March 2020. Click here for more information on this month's prizes!
Number of posts found:
38
