New collaboration with the Center for Open Science

We are super excited to announce that the repliCATS project will work with the Center for Open Science on a new grant funded by the Robert Wood Johnson Foundation (RWJF), extending the work we started on the DARPA SCORE Program.

The SCORE program demonstrated the potential of using algorithms to evaluate claims on a large scale efficiently. Under the RWJF’s grant, COS, in partnership the repliCATS project and The Pennsylvania State University, will advance the development of these efforts in four ways:

• Expanding beyond the core social and behavioral science disciplines covered by SCORE by adding assessment of health research. Extending the program into new disciplines will help establish how generalizable the emerging algorithm solutions can be and will engage more research communities in addressing concerns about research credibility.
• Increasing the number of algorithms providing credibility assessments through a public competition. SCORE involved four algorithmic approaches that generated independent credibility assessments. Expanding to a broader range of algorithm solutions will strengthen the overall accuracy of predictions and decrease the risk of algorithmic bias by developing many more approaches than the original program generated.
• Prototyping algorithmic assessment of claims within the research community to learn how researchers perceive automated scoring of research credibility and what could make it most useful to this community.
• Conducting research and user-testing on the best ways to convey algorithm scores to encourage appropriate use and mitigate risks.

The repliCATS project will continue to crowdsource group judgements of credibility. Professor Fiona Fidler, the chief investigator of the repliCATS project and co-director of MetaMelb, at the University of Melbourne, noted, “SCORE is creating new efforts that are an exciting opportunity to extend our research methodology and advancing dataset about confidence assessment that captures the diversity and depth of how humans reason about and judge evidence.”

You can read more about the wider research program here, https://www.cos.io/about/news/cos-expands-score-program-efforts