Author Guidelines for Verification Reports

Building upon the data we have already published in JOPD, we now accept Verification Reports as an attempt to validate the extent to which these datasets are accessible, reproducible, and valuable.

Verification Reports are a form of empirical article in which authors evaluate the claims in published research through reanalysis of the original study data. The purpose of this format is to assess the credibility of previous conclusions by repeating the original analyses to test computational reproducibility and/or by reporting new analyses to test robustness.

Verification Reports are designed to focus on the same (or closely related) claims as the original study using the same data. They are not appropriate for asking a completely different question of the same data, or for asking the same question using new data. Submissions that report tests of both reproducibility and robustness are preferred, and tests of robustness should always be accompanied by tests of reproducibility.

In some cases, authors may be uncertain whether robustness checks will be judged to be asking a “completely different question” compared with the original study and therefore judged ineligible. The definition of a “completely different question” is necessarily subjective and will be assessed by editors on a case-by-case basis, but a reasonable heuristic to suggest suitability is if a positive answer could be given to the following questions: (a) could the robustness checks potentially strengthen or weaken the original conclusions? (b) in the case of hypothesis-driven research, will the re-analysis test the same hypotheses as the original study? Authors are welcome to contact the editorial office for presubmission advice about specific scenarios.

Submission eligibility

All authors of a Verification Report must be independent of the original study and its authors. In practical terms, this means they cannot be authors or co-authors of the original work. They also cannot be active collaborators (e.g., holding grant funding in the previous three years or other close connections) with any of the original authors. Having shared publications as a result of large collaborative projects where direct contact has been minimal is acceptable but should be flagged during submission. Submitting authors are asked to confirm this condition as part of the submission process.

By default, only empirical articles based upon data published in the Journal of Open Psychology Data currently fall within scope, although this does not guarantee acceptance (see review criteria below). The editors will not consider submissions focusing on research published in other journals at this moment. Where the submitting authors have any questions about eligibility, we recommend submitting a presubmission enquiry.

Verification Reports are not subject to a word limit but should be written as concisely as possible.

Submissions

Verification Reports are assessed according to the value of the question and quality of the method and (re)analysis. Whether the results of the (re)analysis confirm or disconfirm the claims of the original study will be irrelevant to editorial decisions.

The review process takes place over two stages. Authors initially submit a Stage 1 manuscript including only an Abstract, Introduction and Method, with any results known to the authors temporarily redacted. The Stage 1 submission, including the Abstract, must also avoid anticipating the findings or conclusions. The Stage 1 manuscript should include a brief introduction to the topic, a clear justification of the importance of the verification attempt, and a detailed protocol describing the (re)analyses. As the data is openly available, the submitting authors have the option to either (a) submit prior to analysing the data (with the Stage 1 manuscript and cover letter noting that data have not yet been analysed), or (b) to submit the Stage 1 manuscript after completing the (re)analyses but ensuring that all mention of the results is redacted, including the Results and Discussion sections in their entirety.

The Stage 1 manuscript should be accompanied by a cover letter that includes the following:

• A brief case for the importance of the verification attempt, independently of the results
• A statement affirming (a) that none of the authors of the Verification Report were authors or co-authors of the original work, and (b) that that none of the authors of the Verification Report are active collaborators of any of the authors of the original work (including holding shared grant funding in the previous three years or any other close connections). Having shared publications as a result of large collaborative projects where direct contact has been minimal is acceptable but should be flagged during submission. Where authors are in doubt about the eligibility of a submission on these grounds, please contact the editorial office in advance of submission.
• A statement making clear whether the authors are submitting their Stage 1 manuscript prior to acquiring the data OR instead whether they have already acquired the data and completed the (re)analyses but with the results redacted.

Stage 1 Verification Reports are assessed according to the following criteria:

1. The value of the verification attempt. The editors will assess the importance of the verification attempt prior to in-depth review. The value of a Stage 1 manuscript may be diminished where the editors or reviewers judge that the original claims have already been subjected to sufficient evaluation (e.g. through prior Verification Reports or other means) and a more recent replication would serve as a more informative target for verification. Verification attempts that take into account the full basis of published claims are also more likely to be accepted. For example, where the conclusions of the original article were based on multiple studies within one article, a Verification Report may be stronger where it assesses the reproducibility and/or robustness of all studies in the article rather than a subset. Authors are also welcome to submit a single Verification Report scrutinising multiple studies and datasets e.g., where the data has been used for multiple, similar, publications. Submissions that propose tests of both reproducibility and robustness will receive higher priority than either reproducibility or robustness alone. Submissions that propose tests of robustness without also proposing tests of reproducibility are unlikely to satisfy this criterion. Editors and reviewers will assess the value of the verification attempt independently of the results.
2. The methodological validity of the reported (re)analyses. All (re)analyses must be clearly justified, statistically defensible, and designed to interrogate either the reproducibility of the original analyses or the reproducibility of the original analyses plus robustness of the original claims using new analyses. In general, submissions that seek to use the original data to answer different questions from the original study will be desk rejected.

Manuscripts that are reviewed favourably at Stage 1 will receive an in-principle acceptance (IPA), similar to the policy for Registered Reports. IPA commits the journal to publishing the completed verification attempt regardless of the eventual results, provided the submitting authors adhere to their approved protocol and that the conclusions are based on the evidence obtained. Following IPA, authors are required to publicly register their Stage 1 manuscript at http://osf.io/rr (instructions on how to do so will be contained in the Stage 1 acceptance letter). Authors can then resubmit a Stage 2 manuscript that includes the results of the verification attempt plus a Discussion and Conclusion.

Stage 2 Verification Reports are assessed according to the following criteria:

1. Adherence to protocol. Authors must follow the protocol approved at Stage 1. In cases where the results were unknown to the submitting authors until after Stage 1 acceptance, deviations from protocol (including altered or additional analyses) may be permitted to account for unanticipated characteristics of the now-observed data. Any such deviations must be transparently flagged and clearly justified.
2. The extent to which the conclusions of the (re)analyses are based on the evidence obtained. The submitting authors must present an appropriately balanced and unbiased interpretation of the results that takes into account limitations of both the original study and the (re)analyses.

At Stage 2, the abstract should make clear whether or not the original claims were verified by the (re)analyses, and the title (if relevant) can also be updated. The Stage 2 manuscript (main text) must include a direct URL to the registered Stage 1 manuscript, using one of the following statements:

• If the authors had knowledge of the results of the (re)analyses prior to IPA, then the following statement should be included: “This article received results-blind in-principle acceptance (IPA) on [date]. Following IPA, the accepted Stage 1 version of the manuscript, not including results and discussion, was preregistered on the OSF [URL TO PUBLIC PROTOCOL]. This preregistration was performed after data analysis.”
• If the authors had not yet analysed the data prior to IPA, then the following statement should be included: “This article received results-blind in-principle acceptance (IPA) on [date]. Following IPA, the accepted Stage 1 version of the manuscript, not including results and discussion, was preregistered on the OSF [URL TO PUBLIC PROTOCOL]. This preregistration was performed prior to data analysis.”

Submission Templates

You can use one of the paper templates below to prepare your manuscript.

Stage 1 Verification Report Submission Template

Stage 2 Verification Report Submission Template

Verification Report Workflow

Click to download a flow chart explaining the submission and review process for verification reports.

Reviewer selection

Judging the importance and validity of a reanalysis can benefit from intimate knowledge of the original data; therefore the authors of the original study may be invited to review a Verification Report even when they have had prior contact with the submitting authors. Authors may request that the original authors be excluded from the review process, however this may not always be possible to guarantee. However, in assessing the reviews, the editors will take into account the risk that authors of the original work may have a vested interest in a particular outcome being accepted or rejected. 

Permissions

The author is responsible for obtaining all permissions required prior to submission of the manuscript. Permission and owner details should be mentioned for all third-party content included in the submission or used in the research.

If a method or tool is introduced in the study, including software, questionnaires, and scales, the license this is available under and any requirement for permission for use should be stated. If an existing method or tool is used in the research, it is the author's responsibility to check the license and obtain the necessary permissions. Statements confirming that permission was granted should be included in the Materials and Methods section.

Start Submission