If you missed it, Dan Quintana and James Heathers who run the very awesome (we think) Everything Hertz podcast recently interviewed our chief investigator Fiona Fidler live. They talk about:
- The story behind repliCATS
- Australia's best export, Tim Tams, and
- The SCORE project organised by DARPA amongst other things.
The episode is here:
Episode citation and permanent link:
Quintana, D.S., Heathers, J.A.J. (Hosts). (2019, October 21) "Predicting the replicability of research ", Everything Hertz [Audio podcast], DOI: 10.17605/OSF.IO/KZPYG, Retrieved from https://osf.io/kzpyg/
The repliCATS team have had a very busy November 2019!
First, we ran a 100-person workshop before the inaugural Association for Interdisciplinary Meta-research and Open Science on 6 November 2019 at the University of Melbourne.
Of course, there were tim tams, prizes and a whole lot of energy at the pre-AIMOS workshop.
https://twitter.com/sdtokac/status/1192409702765694977?s=20
Then we were in Wellington (New Zealand) where we ran a smaller workshop with the lovely folk over in psychology the following week!
https://twitter.com/replicats/status/1196604406227095552?s=20
What's next for the repliCATS team?
We're busy tweaking our platform, and in early December 2019 we plan to launch a fully remote platform! This means groups of people can participate virtually from anywhere in the world - this could be as individuals looking for claims they are interested in assessing to journal clubs or discussion groups picking claims to work on together.
Want to find out more? Get in touch via repliCATS-project@unimelb.edu.au
We will be running an all-day workshop on 6 November 2019 at the University of Melbourne.
There are 55 travel grants of US$400 available (for participants outside of metropolitan Melbourne) and free registration to the first Association for Interdisciplinary Meta-research & Open Science (AIMOS) conference is included (for all participants).
Apply now to attend this full day workshop.
What will you do at the workshop?
The repliCATS project aims to estimate the replicability of published research claims in the social and behavioural science.
Working in small groups of 5-6, you will be asked to evaluate approximately a dozen social and behavioural science research claims. Each group will have its own facilitator, who will guide you through a structured elicitation protocol.
The workshop will be fully catered.
Why should you participate?
We recently ran a workshop in Rotterdam where we assessed 575 claims over two days! The 200+ workshop participants told us that they appreciated the opportunity to consider and evaluate many different scientific claims and gain exposure to a variety of research designs and approaches. Our hope is that your participation in the workshop will build/enhance the way you think critically about the evidence you are evaluating.
More information about the travel grants
There are 55 travel grants available to participants who live outside of metropolitan Melbourne (including remote Victoria, interstate and overseas). The amount to be paid will be USD 400 (~AUD 580). The travel award is conditional on your participation prior to and during the workshop. These grants also include free registration for the AIMOS conference. Further details about the travel grants can be found here.
Have questions?
You can contact Raquel Ashton by e-mailing repliCATS-project@unimelb.edu.au
Hannah Fraser is the repliCATS research coordinator. In a project about gathering experts and facilitating their ability to assess the replicability of research, she sees her role as facilitating the efforts of the repliCATS team. A meta-facilitator for meta-research.
Hannah completed her PhD in the Quantitative and Applied Ecology group at the University of Melbourne and is passionate about improving the rigor, generalisability and usefulness of ecological research. This began as an exploration of the way researchers study woodland birds (birds that live and/or breed in Australia’s vanishing woodland habitats) and, inspired by FIona Fidler and the meta-researchers from psychology, medicine and economics, has now branched into studying reproducibility and replicability.
Fortunately, this makes her an excellent fit for the repliCATS project.
Aside from the small task of facilitating experts’ assessments of 3,000 research claims for the repliCATS project, Hannah’s research interests include:
- Questionable research practices - identity, incidence, implications
- Researchers views on replication studies - value, venues, vagaries
- Many Analysts one ecology dataset, one environmental biology dataset - duplication, differences, divergence
- Open science innovations - effectiveness, efficiency, endorsement.
Some of Hannah’s best thinking happens on long walks with colleagues, friends, family and dogs. To relax, she gets into home improvement (just dont ask her to paint your house) and loves to play board games with other aficionados.
Her top 3 board games are:
- Wingspan (a board game about birds - enough said)
- Small World (berserker orcs, yes please!)
- Sushi go party (because it’s fast and silly)
--
Hannah also discussed in Simine Vazire's blog, sometimes i'm wrong, discussing the challenges facing ecology: https://sometimesimwrong.typepad.com/wrong/2019/07/status-part-i.html
You can follow Hannah on twitter @HannahSFraser
The first repliCATS elicitation workshop was held in Rotterdam on 5-6 July 2019, before the annual Society for the Improvement of Psychological Sciences (SIPS) conference. We are so grateful to the SIPS2019 organisers for accomodating us!
This workshop was the first test of our custom platform, as well as the deployment of the IDEA protocol to crowdsource assessments about the replicability of research claims in the social and behavioural sciences.
Here are some workshop statistics:
- we had 156 participants and 30 facilitators working in groups of 5-6 people over the two days
- at the end of which we assessed 575 (out of 3,000) research claims in psychology, marketing and experimental economics...
- ...that amounted to over 27,000 quantitative data points and 12,072 justifications or qualitative reasoning,
- and in doing so, we consumed x32 packets of tim tams.
There were also some prizes!
What's next for the repliCATS project?
The next big workshop for the repliCATS project will be held at the University of Melbourne on 5 November 2019, before the inaugural conference for the Association for Interdisciplinary Meta-research & Open Science (AIMOS).
There are x55 travel grants of US$450.00 available to subsidise participants from outside metropolitan Melbourne, interstate or overseas.
To find out more, and to express interest, visit: https://www.aimos2019conference.com.
Media release | Wednesday, 3 April 2019
In response to the 'replication crisis' in a number of scientific fields, a major international research program seeks to develop Artificial Intelligence to help evaluate the credibility of scientific evidence we use to make decisions.
The University of Melbourne is currently the only Australian team selected by the US Government’s Defense Advanced Research Project Agency (DARPA) to work on the “Systematizing Confidence in Open Research and Evidence” or SCORE program.
With up to US$6.5m million in funding, the University’s team, Collaborative Assessment for Trustworthy Science or the repliCATS project, will assess the replicability of thousands of social and behavioural research claims.
This work will inform the AI component of SCORE. It will also help us understand how to recognise credible research.
The University’s team plans to crowdsource thousands of experts – working in psychology, sociology, criminology, economics, business, marketing, political science and education – to meet the SCORE challenge.
Associate Professor Fiona Fidler who is a reproducibility and Open Science expert will lead the University of Melbourne team. She said: “This is by far the most ambitious reproducibility project the social and behavioural sciences have seen.
“It will be a defining moment in how we understand the evidence base in the published literature of those fields.”
For the full release, visit: https://about.unimelb.edu.au/newsroom/home.
--
Media enquiries: Louise Bennet | 0412 975 350 | e.bennet@unimelb.edu.au
Friday 5 April | 1-3pm | Forum theatre, Arts West
Join an interdisciplinary panel of local and international speakers to discuss the role of ethics in reforming scientific practice.
Inappropriate statistical practices, poor research design, and other questionable research practices has led to the ‘replication crisis’ in many scientific disciplines. The challenge we face is how to effect change – how to sustain the credibility revolution. The panellists will first explore the real consequences and damage caused by the replication crisis. Visiting scholars Anne Scheel (Eindhoven University of Technology) and Michèle Nuitjen (Tilburg University) will then open the discussion, which will centre on the role of ethics in reforming scientific practice – from how we train scientists to how we regulate their work.
Speakers:
View and share the event flyer.
Brian Nosek is the co-Founder and Executive Director of the Center for Open Science (http://cos.io/) that operates the Open Science Framework (http://osf.io/). COS is enabling open and reproducible research practices worldwide.
Brian will be giving a free public lecture on Thursday, 4 April at the University of Melbourne on the gap between scientific values and practice.
You can read the abstract and register to attend here.
Number of posts found:
38
