Building upon the data we have already published in JOPD, we now accept Verification Reports as an attempt to validate the extent to which these datasets are accessible, reproducible, and valuable.
Verification Reports are a form of empirical article in which authors evaluate the claims in published research through reanalysis of the original study data. The purpose of this format is to assess the credibility of previous conclusions by repeating the original analyses to test computational reproducibility and/or by reporting new analyses to test robustness.
Verification Reports are designed to focus on the same (or closely related) claims as the original study using the same data. They are not appropriate for asking a completely different question of the same data, or for asking the same question using new data. Submissions that report tests of both reproducibility and robustness are preferred, and tests of robustness should always be accompanied by tests of reproducibility.
In some cases, authors may be uncertain whether robustness checks will be judged to be asking a “completely different question” compared with the original study and therefore judged ineligible. The definition of a “completely different question” is necessarily subjective and will be assessed by editors on a case-by-case basis, but a reasonable heuristic to suggest suitability is if a positive answer could be given to the following questions: (a) could the robustness checks potentially strengthen or weaken the original conclusions? (b) in the case of hypothesis-driven research, will the re-analysis test the same hypotheses as the original study? Authors are welcome to contact the editorial office for presubmission advice about specific scenarios.
All authors of a Verification Report must be independent of the original study and its authors. In practical terms, this means they cannot be authors or co-authors of the original work. They also cannot be active collaborators (e.g., holding grant funding in the previous three years or other close connections) with any of the original authors. Having shared publications as a result of large collaborative projects where direct contact has been minimal is acceptable but should be flagged during submission. Submitting authors are asked to confirm this condition as part of the submission process.
By default, only empirical articles based upon data published in the Journal of Open Psychology Data currently fall within scope, although this does not guarantee acceptance (see review criteria below). The editors will not consider submissions focusing on research published in other journals at this moment. Where the submitting authors have any questions about eligibility, we recommend submitting a presubmission enquiry.
Verification Reports are not subject to a word limit but should be written as concisely as possible.
Verification Reports are assessed according to the value of the question and quality of the method and (re)analysis. Whether the results of the (re)analysis confirm or disconfirm the claims of the original study will be irrelevant to editorial decisions.
The review process takes place over two stages. Authors initially submit a Stage 1 manuscript including only an Abstract, Introduction and Method, with any results known to the authors temporarily redacted. The Stage 1 submission, including the Abstract, must also avoid anticipating the findings or conclusions. The Stage 1 manuscript should include a brief introduction to the topic, a clear justification of the importance of the verification attempt, and a detailed protocol describing the (re)analyses. As the data is openly available, the submitting authors have the option to either (a) submit prior to analysing the data (with the Stage 1 manuscript and cover letter noting that data have not yet been analysed), or (b) to submit the Stage 1 manuscript after completing the (re)analyses but ensuring that all mention of the results is redacted, including the Results and Discussion sections in their entirety.
The Stage 1 manuscript should be accompanied by a cover letter that includes the following:
Stage 1 Verification Reports are assessed according to the following criteria:
Manuscripts that are reviewed favourably at Stage 1 will receive an in-principle acceptance (IPA), similar to the policy for Registered Reports. IPA commits the journal to publishing the completed verification attempt regardless of the eventual results, provided the submitting authors adhere to their approved protocol and that the conclusions are based on the evidence obtained. Following IPA, authors are required to publicly register their Stage 1 manuscript at http://osf.io/rr (instructions on how to do so will be contained in the Stage 1 acceptance letter). Authors can then resubmit a Stage 2 manuscript that includes the results of the verification attempt plus a Discussion and Conclusion.
Stage 2 Verification Reports are assessed according to the following criteria:
At Stage 2, the abstract should make clear whether or not the original claims were verified by the (re)analyses, and the title (if relevant) can also be updated. The Stage 2 manuscript (main text) must include a direct URL to the registered Stage 1 manuscript, using one of the following statements:
You can use one of the paper templates below to prepare your manuscript.
Stage 1 Verification Report Submission Template
Stage 2 Verification Report Submission Template
Judging the importance and validity of a reanalysis can benefit from intimate knowledge of the original data; therefore the authors of the original study may be invited to review a Verification Report even when they have had prior contact with the submitting authors. Authors may request that the original authors be excluded from the review process, however this may not always be possible to guarantee. However, in assessing the reviews, the editors will take into account the risk that authors of the original work may have a vested interest in a particular outcome being accepted or rejected.
The author is responsible for obtaining all permissions required prior to submission of the manuscript. Permission and owner details should be mentioned for all third-party content included in the submission or used in the research.
If a method or tool is introduced in the study, including software, questionnaires, and scales, the license this is available under and any requirement for permission for use should be stated. If an existing method or tool is used in the research, it is the author's responsibility to check the license and obtain the necessary permissions. Statements confirming that permission was granted should be included in the Materials and Methods section.