Building upon the data we have already published in JOPD, we now accept Verification Reports as an attempt to validate the extent to which these datasets, and indeed those found in the wider body of evidence, are accessible, reproducible, and valuable.
Verification Reports are a form of empirical article in which authors evaluate the claims in published research through reanalysis of the original study data. The purpose of this format is to assess the credibility of previous conclusions by repeating the original analyses to test computational reproducibility and/or by reporting new analyses to test robustness.
Verification Reports are designed to focus on the same (or closely related) claims as the original study using the same data. They are not appropriate for asking a completely different question of the same data, or for asking the same question using new data. Submissions that report tests of both reproducibility and robustness are preferred, and tests of robustness should always be accompanied by tests of reproducibility.
In some cases, authors may be uncertain whether robustness checks will be judged to be asking a “completely different question” compared with the original study and therefore judged ineligible. The definition of a “completely different question” is necessarily subjective and will be assessed by editors on a case-by-case basis, but a reasonable heuristic to suggest suitability is if a positive answer could be given to the following questions: (a) could the robustness checks potentially strengthen or weaken the original conclusions? (b) in the case of hypothesis-driven research, will the re-analysis test the same hypotheses as the original study? Authors are welcome to contact the editorial office for presubmission advice about specific scenarios.
All authors of a Verification Report must be independent of the original study and its authors. In practical terms, this means they cannot be authors or co-authors of the original work. They also cannot be active collaborators (e.g., holding grant funding in the previous three years or other close connections) with any of the original authors. Having shared publications as a result of large collaborative projects where direct contact has been minimal is acceptable but should be flagged during submission. Submitting authors are asked to confirm this condition as part of the submission process.
By default, all empirical articles published in the JOPD fall within the disciplinary scope of Verification Reports, although this does not guarantee acceptance. The editors will also happily consider submissions focusing on research published in other journals and will not discriminate based upon origin. Where the submitting authors have any questions about eligibility, or plan to target a study published in another journal, we recommend submitting a presubmission enquiry. Where the submitting authors intend to investigate multiple previous studies (e.g. to investigate several articles within the same field), authors should combine their attempts into a single manuscript.
Verification Reports are not subject to a word limit but should be written as concisely as possible.
A Verification Report can be published only in cases where the submitting authors have either obtained the original study data or have exhausted all reasonable attempts to obtain them. The definition of what constitutes “reasonable” will be judged on a case-by-case basis by the editors, but as general guidance, attempts to obtain data may be considered reasonably exhausted where the authors of the Verification Report have received an explicit refusal from the data owners/curators to share the data, where attempts to communicate with the data owners/curators through all available channels (email, telephone) have repeatedly failed, where the data owners/curators are either deceased or no longer exist, or where the data are confirmed as being lost or destroyed.
In cases where the data are already in the public domain, authors can usually submit a Verification Report without needing to contact the original authors, provided any public datasets are cited in accordance with Standard #1 of the journal’s Transparency and Openness Promotion (TOP) guidelines. In cases where the data are not in the public domain, the submitting authors should request the necessary data and analysis code from the original authors or relevant data custodian, noting the intention to submit a Verification Report, and stating that according to the policy at JOPD, the authors of the original study are not eligible to serve as co-authors of the report. Submitting authors are responsible for ensuring that they adhere to any legal or ethical restrictions in the handling, distribution, and (re)analysis of the data and code. The legal and ethical conditions of reuse, and any restrictions therein, must be noted in the published Verification Report in accordance with the TOP guidelines policy. Submitting authors have two options in cases where sufficient data and/or code to perform complete tests of reproducibility or robustness cannot be obtained from the original authors. The first option is to submit a normal Verification Report if sufficient data/code are available to permit partial but informative testing. Where none of the original data or code can be retrieved, a Verification Report may be possible if the report is instead based on analyses of summary statistics in the original article (e.g. using tools such as GRIM, GRIMMER, SPRITE, statcheck, or other methods).
Verification Reports are assessed according to the value of the question and quality of the method and (re)analysis. Whether the results of the (re)analysis confirm or disconfirm the claims of the original study will be irrelevant to editorial decisions.
The review process takes place over two stages. Authors initially submit a Stage 1 manuscript including only an Abstract, Introduction and Method, with any results known to the authors temporarily redacted. The Stage 1 submission, including the Abstract, must also avoid anticipating the findings or conclusions. The Stage 1 manuscript should include a brief introduction to the topic, a clear justification of the importance of the verification attempt, and a detailed protocol describing the (re)analyses. In cases where the study data are not publicly available, authors can submit their Stage 1 manuscript prior to analysing the data, provided they have secured the necessary agreement from the original authors to make the data available. In cases where the submitting authors are already in possession of the data (with permissions secured from the appropriate data owners/curators) or it is publicly accessible, they have the option to either (a) submit prior to analysing the data (with the Stage 1 manuscript and cover letter noting that data have not yet been analysed), or (b) to submit the Stage 1 manuscript after completing the (re)analyses but ensuring that all mention of the results is redacted, including the Results and Discussion sections in their entirety.
The Stage 1 manuscript should be accompanied by a cover letter that includes the following:
Stage 1 Verification Reports are assessed according to the following criteria:
Manuscripts that are reviewed favourably at Stage 1 will receive an in-principle acceptance (IPA), similar to the policy for Registered Reports. IPA commits the journal to publishing the completed verification attempt regardless of the eventual results, provided the submitting authors adhere to their approved protocol and that the conclusions are based on the evidence obtained. Following IPA, authors are required to publicly register their Stage 1 manuscript at http://osf.io/rr (instructions on how to do so will be contained in the Stage 1 acceptance letter). Authors can then resubmit a Stage 2 manuscript that includes the results of the verification attempt plus a Discussion and Conclusion.
Stage 2 Verification Reports are assessed according to the following criteria:
At Stage 2, the abstract should make clear whether or not the original claims were verified by the (re)analyses, and the title (if relevant) can also be updated. The Stage 2 manuscript (main text) must include a direct URL to the registered Stage 1 manuscript, using one of the following statements:
You can use one of the paper templates below to prepare your manuscript.
Stage 1 Verification Report Submission Template
Stage 2 Verification Report Submission Template
Judging the importance and validity of a reanalysis can benefit from intimate knowledge of the original data; therefore the authors of the original study may be invited to review a Verification Report even when they have had prior contact with the submitting authors. Authors may request that the original authors be excluded from the review process, however this may not always be possible to guarantee. However, in assessing the reviews, the editors will take into account the risk that authors of the original work may have a vested interest in a particular outcome being accepted or rejected.
The author is responsible for obtaining all permissions required prior to submission of the manuscript. Permission and owner details should be mentioned for all third-party content included in the submission or used in the research.
If a method or tool is introduced in the study, including software, questionnaires, and scales, the license this is available under and any requirement for permission for use should be stated. If an existing method or tool is used in the research, it is the author's responsibility to check the license and obtain the necessary permissions. Statements confirming that permission was granted should be included in the Materials and Methods section.