Resources — SMART preprints

Question 1: How well do you understand this claim?

Purpose: To understand if anything affects your ability to interpret the specific claim. 

Clarification: We know that scientific papers vary in clarity and comprehensibility. It’s possible that a paper:

• is vague;
• is poorly written;
• relies on an unfamiliar procedure;
• contains too much jargon;
• is unclear about exactly what claim is or how it is interpreted;
• is about a concept that you are not familiar with and/or have difficulty conceptualising.

These factors can all contribute to your ability to be able to interpret the claim and may in turn lead to different interpretations by the group. There is a comments box below this question, where you can restate the specific claim in your own words. If there are particular terms or concepts that you have difficulty interpreting, you can list them in this text box as well. Please do not put any discussion of your assessment of the claim in this box. Put all of your reasoning in the final text box, below Question 7: Calibration.

Answering the Comprehensibility question:

We’re asking for this on a scale of 0 to 100. 0 means that you have no idea what the claim means, 100 means it’s perfectly clear to you. This is not an objective measurement – try your best to estimate a number for comprehensibility, and just try to be consistent between papers.

Some papers may be outside of your main fields or use words that you are unfamiliar with. This might cause you to immediately put ‘I have no idea what the claim means’. However, with a little bit of effort, you can usually deduce what is being asked. If after your reading of the paper you still cannot work it out then you should definitely indicate this to us and consider whether that is indicating something about the quality of the research being described.

Tooltip: We know the clarity and comprehensibility of scientific papers varies. We are interested in your honest account of how well you understand this paper and its central claim.

Comments box: We have provided a comment box below the question so you can try to rephrase what you think the claim is about. This will be useful in the Discussion phase to prompt your memory about your initial interpretation of the question and to understand other participants’ interpretations. Please do not put any discussion of your assessment of the claim in this box.

The placeholder text for this textbox is: The clarity and comprehensibility of scientific papers varies. Tell us how well you understand this claim.

Question 2: How consistent is this claim with your existing beliefs?

Purpose: To capture your assessment of the prior plausibility of the claim.

Clarification: Sometimes we hear a claim in a paper and have a strong feeling that it is or is not very plausible either within the context of the experimental design, or more broadly (i.e. relating to a relationship that would generalize across contexts or experimental designs).

These prior beliefs can be useful. We’ve included this question here to allow you to state your prior belief about whether the claim describes a real effect, without considering the design of or the evidence from this particular study.

Don’t spend too much time on this question. In following questions, you will examine the evidence this particular study provides for the claim more critically.

Answering the Plausibility question:

We’re asking for this on a scale of 0 to 100. 0 means that the claim is exactly contrary to your pre-existing beliefs, 100 means it’s perfectly compatible with them. As with Question 1: Comprehensibility, estimate a number and just try to be consistent between papers.

The word ‘plausible’ means different things to different people. For some people, almost everything is ‘plausible’, while other people have a stricter interpretation. Don’t be too focused on the precise meaning of ‘plausible’ – you could also consider words like ‘possible’ or ‘realistic’ here. Try to let us know if some claims are clearly more plausible (or implausible) than others.

Tooltip: How likely would you have been to accept this claim before you considered the evidence from this paper?

Question 3: What is the probability that an independent close replication of this claim would find results consistent with the original paper?

Purpose: The question is asking about a close replication of the specific claim.

Clarification: A close replication is a new experiment that follows the methods of the original study with a high degree of similarity, varying only aspects where there is a high degree of confidence that they are not relevant to the research claim. People often use the term direct replication – however, no replication is perfectly direct, and decisions always need to be made about how to perform replications. To answer this question, you may want to imagine what kinds of decisions you would face if you were asked to replicate this claim, and to consider the effects of making different choices for these decisions.
For this purpose, we consider “consistent with the original paper” to mean a statistically significant effect (defined with an alpha of 0.05) in the same direction as the original study, using the same statistical technique as the original study.

Answering the Replicability question:

In this question, we want you to try and think of reasons why the specific claim may or may not replicate. We understand that your thoughts about the prior plausibility of this claim are likely to influence your judgement regarding this question. However, we’d like you to try and think more critically of other reasons why this particular study may (or may not) replicate, independently of whether the described effect is a real one.

For close replications involving new data collection, we would like you to imagine 100 (hypothetical) new replications of the original study, combined to produce a single, overall replication estimate (i.e., a good-faith meta-analysis with no publication bias). Assume that all such studies have both a sample size that is at least as large as the original study and high power (90% power to detect an effect 50-75% of the original effect size with alpha=0.05, two-sided).

Sometimes it is clear that a close replication of a claim involving new data collection is impossible, or infeasible. In these cases, you should think of data analytic replications, in which the central claim is tested against another pre-existing dataset that provides a fair test. Again, imagine 100 datasets analysed with results are combined to produce a single, overall replication estimate.

There are many things you might consider when making your judgement. The IDEA protocol operates well when a diversity of approaches is combined. There is no single ‘correct’ checklist of things to assess. However, some things you may wish to consider include statistical details, experimental design and contextual information about how the study was undertaken.

For replicability, we ask you to provide a lower bound, upper bound and best estimate. For further details see: The three-step response format.

Tooltip: Assume replications are competent and done in good faith. A close replication is a high-powered collection of new data to test an original study, without altering important features of the study design. Sometimes no such attempt is feasible. In these cases, replications will test claims with matching analyses on similar, pre-existing datasets. Check training materials for details on close replications and data-analytic replications, including how we define “consistent with the original paper” and high statistical power.

Question 4: What is the probability that an independent analysis of the same data would find results consistent with the original claim?

Purpose: To capture your beliefs about the analytic robustness of the claim.

Clarification: The term robustness is used here to represent the stability and reliability of a research finding, given the data. Is the claim likely to be due to an idiosyncratic choice of analytic strategy or not? 

Answering the Robustness question:

It might help to think about it in this way: imagine 100 analysts received the original data and devised their own means of investigating the claim (e.g using a different, but appropriate statistical model or technique), how many would find a statistically significant effect in the same direction as the original finding? You should assume that any such analyses are conducted competently and in good faith.

Like Q3: Replicability, we ask you to provide a lower bound, upper bound and best estimate. For further details see: The three-step response format.

Tooltip: Assume all potential analyses are competent and done in good faith. Check training materials for details on how to assess “consistent with the original paper”.

This question has two parts, asking you to give an overall rating, and identify the features of the study that affect its generalizability.

Question 5a: Going beyond close replications and reanalyses, how well would this claim generalize to other ways of researching the question?

Purpose: The question is asking about whether the specific claim would hold up or not under different ways of studying the question. 

Clarification: This question is asking about generalizations of the original study or ‘conceptual replications’. We want you to consider generalizations across all relevant features, such as the particular instruments or measures used, the sample or population studied, and the time and place of the study.

Answering the Generalizability (Rating) question:

This question is asked on a scale of 0 to 100, where 0 means the claim is not at all generalizable and 100 means it is completely generalizable.

We know that there are many different features of a given study that potentially limit generalizability, and they may have different levels of concern, so it might be tricky to work out a single rating across all of them. Assess this in whatever way seems best.

You might want to imagine 100 (hypothetical) conceptual replications of the original study, each of which varies just one specific aspect of the study design (e.g. sample, location, operationalisation of a variable of interest,…) while holding everything else constant. Across this hypothetical set of conceptual replications, all relevant aspects of the study are varied in differing ways. In your estimate, how many of these would produce a finding that is consistent with the original study?

Tooltip: We’re asking you to judge whether the results of this claim are due to an idiosyncratic feature of this study, such as the specific sample, setting, time, or the particular instruments or measurements used. How well do you think the result would hold up beyond those limits? 

Question 5b: Please select the feature(s), if any, that you think limit the generalizability of the findings.

Purpose: The question is asking you to list the features of the study that raise substantial generalizability concerns.

Clarification: Select the features for which you definitely have generalizability concerns. Don’t select a feature if you simply think that it is possible that the study will not generalize over that feature. 

Answering the Generalizability (Features) question:

You can select more than one feature – select all features you think raise substantial generalizability concerns.

If there is a feature that you think raises substantial generalizability concerns, but we have not included then select ‘Other’ and briefly list the feature(s) in the text box. Please do not use this text box to discuss why you think there are generalizability concerns or to give specific details about a feature you have selected. If you have any comments or thoughts on this, please write them in the final text box, below Question 7: Calibration.

Tooltip: There are many features of any dataset. A claim might generalize well over some but fail to generalize over others. We want you to list the features that may significantly limit the generalizability of this claim. Do not discuss the reasons for your concerns about generalizability in this text box. Do that in the text box below.

Question 6: How appropriate is the statistical test of this claim?

Purpose: The question is asking about the extent to which a set of statistical inferences, and their underlying assumptions, are appropriate and justified, given the research hypotheses and the (type of) data.

Clarification: This question focuses on the extent to which the statistical models/tests are appropriate for testing the research hypotheses. For instance, assumptions may have been violated that would render the chosen test(s) inappropriate, or the statistical model of choice may not be appropriate for the type of data.

Answering the Statistical Validity question:

We’re asking for this on a scale of 0 to 100. 0 means that the statistical analyses are not at all appropriate to test the research hypotheses, 100 means they are entirely appropriate to test the research hypotheses. Again, try to estimate a number and be consistent in the criteria you apply when assessing statistical validity between papers.

Tooltip: By “appropriate” we mean that the correct test for the design is done, precautions are taken to minimize risk of bias and error, and assumptions are reasonable.

Question 7: How reasonable and well-calibrated are the conclusions drawn from the findings in this paper?

Purpose: The question is asking about the extent to which the conclusions drawn from this claim are warranted.

Clarification: This question relates to the stated interpretation of the specific claim, whether the conclusions match the evidence presented, and the limitations of the study. Sometimes a paper’s conclusion(s) might extend beyond what is indicated by the results reported. 

Answering the Calibration question:

This question uses the same 3-point format as Question 3: Replicability and you should assess it in the same way. For further details see: The three-step response format.

Tooltip: By “well-calibrated” we mean that the conclusions match the evidence presented, without overstepping the mark.

Comments box: We have provided a text box below the question. Please use the text box for this Question 7: Calibration to capture all of your thinking about your assessment of this paper. For further details see Providing a rationale.

We encourage you to review others’ assessments and – in particular – examine any comments in the final text box below Question 7: Calibration, because this is where assessors capture their thoughts and reasoning on the various dimensions of the paper’s credibility. You can use the comments feature on the platform to react to other participants’ judgements, interrogate their reasoning, and ask questions.

Once again, it is important that you do not use any participant names in comments on the platform, not even your own. Comments will eventually be made public, and we need to keep these anonymous. If you want to refer to other participants, please use the anonymous user name (avatar) they have been assigned e.g. Koala11.

Some ground rules for the Discussion phase, regardless of whether you are leaving comments on the online platform or discussing directly:

• Respect that the group is composed of a diversity of individuals;
• Consider all perspectives in the group. In synchronous discussion, allow an opportunity for everyone to speak;
• Don’t assume everyone has read the same papers or has your skills – explain your reasoning in plain language;
• If someone questions your reasons, they are usually trying to increase their own understanding. Try to provide simple and clear justifications;
• Try to be open-minded about new ideas and evidence.

The following list may be useful to consider when reviewing and commenting on judgements. You do not have to work through these systematically, but consider which may be relevant:

• What did people believe the claim being made was? Was the paper clear? Did everyone understand the information and terms in the same way? If interpretations of the central claim (or any claim) vary, instead of trying to resolve this, focus on discussing what that means for the credibility of the paper.
• Consider the range of estimates in the group and ask questions about extreme values. What would cause someone to provide a high/low estimate for this question?
• Very wide intervals suggest unconfident responses. Are these based on uncertainties of interpretation or are these participants aware of contradictory evidence?
• Very narrow intervals suggest very confident responses. Do those participants have extra information?
• It’s ok if you don’t have good evidence for your beliefs – please feel free to state this.
• If you have changed your mind since your Round 1 estimates it’s good to share this. Actively entertaining counterfactual evidence and beliefs improves your judgements.
• If you disagree with the group that is fine. Please state your true belief when completing Round 2 estimates. This represents the true uncertainty regarding the question, and it should be captured.
• Consider raising counterarguments, not to be a nuisance, but to make sure the group considers the full range of evidence.
• As a group, avoid getting bogged down in details that are not crucial to answering the questions, or in trying to resolve differences in interpretation. Focus on sharing your reasons, not on convincing others of specific question answers.