Background

COVID-19 changed the world in a way that had not been seen in generations (Bashir, Benjiang, & Shahzad, 2020). The COVID virus began circulating in the Wuhan province in China in late 2019 and led to an unprecedented lockdown of the world in March of 2020 (Scherbina, 2021). Within weeks, numerous studies were being published across a variety of domains including medicine, psychology, and allied disciplines (Sommerlad et al., 2021). Some of these studies were revealed to have flaws (Piller & Servick, 2020) whereas other studies quickly defined the world’s response to the virus (Stasi et al., 2020).

In many ways, this pattern mirrors what has previously been being noted in science more generally with the replication crisis (Clegg, & Slaney, 2019). Numerous studies have failed to replicate despite being very influential in the field (Open Science Collaboration, 2012). Although some researchers have speculated that this is an indictment of the field, its publishing practices, and the incentives for publications (Lilienfeld, & Strother, 2020), other researchers have suggested that this is simply science working as it should (Edlund, Cuccolo, Irgens, Wagge, & Zlokovich, 2022).

These dual crises naturally led some researchers to change their focus and to explore the replicability of the early studies being published on the COVID-19 crisis. The two studies detailed in this manuscript are a result of this combined focus.

An initial study (Imhoff & Lamberty, 2020) looked at responses to the COVID-19 pandemic and was focused on whether conspiracy beliefs that the COVID crisis was a hoax would lead to less engagement with containment activities (such as washing hands and physical distancing). Imhoff and Lamberty found that conspiracy beliefs was a strong negative predictor of engaging in mitigation activities such as physical distancing.

Another initial study (Pfattheicher, Nockur, Böhm, Sassenrath & Petersen, 2020), explored whether empathy was related to the motivation for physical distancing. In this study, Pfattheicher and colleagues found a strong correlation between empathy and engaging in physical distancing.

Given the importance of physical distancing and the early publishing of Imhoff and Lamberty, 2020 and Pfattheicher et al. 2020) we decided to attempt a replication of these findings as part of the larger SCORE (Center for Open Science, 2022) project looking at the replicability of the social and behavioral sciences more generally.

Methods

Both studies reported in this paper were part of the larger SCORE project (Center for Open Science, 2022). As a result, all of the studies were pre-registered and went through pre-collection peer review. Additionally, data collection was to be handled in two sequential cross-sectional data collections on the same day. Sample size was based on a priori power calculations: the first round of data collection was calculated to achieve 90% power to detect 75% of the original effect size in the study to be replicated. If the initial sample resulted in non-significant findings, a pooled sample would achieve 90% power to detect 50% of the original effect size. These power calculations were based on the relevant effect size of the focal analysis in the source study (cohen’s g) and were following the guidelines established in the SCORE Project.

The associated files for the Imhoff & Lamberty replication are located here: https://osf.io/9beuv/.

The associated files for the Pfattheicher replication are located here: https://osf.io/nv6a3/.

Study design (Imhoff & Lamberty)

Data was collected from Amazon’s Mechanical Turk (mTurk; limited to participants based in the US). The study was posted with the title “Psychology and COVID-19”. Participants were told that “the study will consist of a handful of demographic and personal questions about themselves and their perceptions of the COVID-19 Situations.” Participants were paid three dollars (USD). Participants were able to read the consent form before participating (which revealed the funder to be the Center for Open Science and the Department of Defense per HSRO regulations). Participants were able to skip any individual item they wanted (per local institution review board [IRB] guidelines).

Initially, participants completed an informed consent document. Next, participants completed the stimuli questions from Imhoff & Lamberty’s preprint (listed below in section 2.5.1). Following this, participants completed a series of demographics related to the COVID-19 crisis and other general demographic variables (also listed in section 2.5.1). Participants also completed an attention check. Finally, participants were debriefed and thanked for their participation.

Also of particular note, participants who completed this data collection were excluded from the Pfattheicher data collection).

Study design (Pfattheicher et al)

Data was collected from Amazon’s mTurk (limited to participants based in the US). The study was posted with the title “Psychology and COVID-19”. Participants were told that “the study will consist of a handful of demographic and personal questions about themselves and their perceptions of the COVID-19 Situations.” Participants were paid three dollars (USD). Participants were able to read the consent form before participating (which revealed the funder to be the Center for Open Science and the Department of Defense per Human Subjects Research Officer [HSRO] regulations). Participants were able to skip any individual item they wanted (per local IRB guidelines).

Initially, participants completed an informed consent document. Next, participants completed the stimuli questions from Pfattheicher et al’s preprint (listed below in section 2.5.2). Following this, participants completed a series of demographics related to the COVID-19 crisis and other general demographic variables (also listed in section 2.521). Participants also completed an attention check. Finally, participants were debriefed and thanked for their participation.

Time of data collection

Both studies reported in this manuscript collected data on August 20th, 2020.

Location of data collection

Both studies reported in this manuscript collected data through Amazon’s mTurk and participants were limited to being United States Citizens. As noted in Huff and Tingley (2015), participants from mTurk skew somewhat more educated and urban than the US population as a whole, but the samples from mTurk are more diverse than the average psychological study. Further, although there is debate in the literature about the quality of data generated by mTurk workers, studies that use protocols similar to the protocols we use find comparable data quality to paid survey panels (Zhang, & Gearhart, 2020).

Sampling, sample and data collection (Imhoff & Lamberty)

104 individuals participated in this study. The demographics included 59 men, 44 women, and one participant who preferred to not answer. The sample was primarily White 71.2%; Black/African American, 9.6%; Asian/Asian American, 7.7%; Hispanic, 9.6%; Other/Multiple, 1.9%, and had a mean age of 35.85 years (SD = 11.24).

Sampling, sample and data collection (Pfattheicher et al)

63 people participated in this study. They consisted of 42 men and 21 women. The sample was primarily White 76.2%; Black/African American, 7.9%; Asian/Asian American, 4.8%; Hispanic, 6.3%; Other/Multiple, 4.8%, and had a mean age of 34.37 years (SD = 11.24).

Materials/Survey instruments (Imhoff & Lamberty)

After completing the consent form, participants completed a series of 17 questions related to their behaviours they may have engaged in response to the COVID pandemic in which they responded to the items on a 7-point Likert style scale (example: “Avoiding social contacts”). Next, on the same response scale, participants responded to six items related to COVID conspiracy beliefs (example: “Dark forces want to use the virus to rule the world”). Next participants indicated their concern about the COVID-19 virus more generally on a seven point unidirectional scale (example: “To what extent are you currently worried about the spread of coronavirus?”). Next, on a seven point Likert-style scale, participants answered 12 questions about general conspiracy beliefs (example: “Most people do not recognize to what extent our life is determined by conspiracies that are concocted in secret”). Finally, participants completed demographics including age, sex, race, zipcode, activities in the last seven days, level of education, household size, household ages, knowledge of COVID-19 more generally, perceptions of threat of COVID-19 more generally, COVID testing experience and diagnosis, chronic disease background, flu vaccine in the 19–20 season, political orientation, as well as an attention check (Please write down what you had yesterday for lunch [1 word is enough]).

Materials/Survey instruments (Pfattheicher et al)

After completing the consent form, participants completed a series of 11 questions related to their behaviours they may have engaged in response to the COVID pandemic in which they responded to the items on a 5-point Likert style scale. These items included measures of empathy (example: I am very concerned about those most vulnerable to coronavirus [COVID-19]) and measures of behavioural changes (example: Because of coronavirus [COVID-19], it is very important that others massively curtail social contact [so-called “social distancing”]). Finally, participants completed demographics including age, sex, race, zipcode, activities in the last seven days, level of education, household size, household ages, knowledge of COVID-19 more generally, perceptions of threat of COVID-19 more generally, COVID testing experience and diagnosis, chronic disease background, flu vaccine in the 19–20 season, political orientation, as well as an attention check (Please write down what you had yesterday for lunch [1 word is enough]).

Quality Control

As noted, attention checks were embedded in the surveys. Additionally, we excluded participants who did not provide responses to the key analysis questions.

Data anonymisation and ethical issues

Given the nature of the project, multiple ethical approvals were obtained. Initial approval was obtained by the Institutional Review Board at the Rochester Institute of Technology (RIT). After RIT granted its approval, the Human Subjects Research Office (HSRO) approval was sought and obtained through the Department of the Army. There was no deception in this study and informed consent was obtained. Additionally, the only identifying information obtained was an mTurk ID which was then deleted after credit was granted.

Existing use of data

To date, no studies have been published using the data reported in this manuscript.

Dataset description and access

Repository location

Imhoff & Lamberty: https://osf.io/9beuv/

Pfattheicher: https://osf.io/nv6a3/

Object/file name

Imhoff and Laberty

Overarching report as part of the SCORE Project: final report imhoff.docx.

Preregistration and peer review of the replication: Imhoff_covid_dPzV-direct replication – edlund – z1K9.

Abstract for IRB: Abstract score 3.docx.

Consent Form: Consent for score 3-revised.docx.

Ethics form (HRPO): HRPO_Protocol)Submission_form (prefilled)-score 3.docx.

RIT IRB Application: Score 3 irb.docx.

Ethics Form: USAMRMC_Proposal_submission_form (prefilled) – score3.docx.

a priori power analysis: POWER_Imhoff_covid_dPzV.zip.

Full materials: Imhoff_covid_dPzV_z1k9_materiasl.pdf.

Data Dictionary: Data dictionary imhoff.docx.

Data (in SPSS): Imhoff_august20,20220_14.42.sav.

Data (in CSV): Imhoff_august20,20220_14.42.csvAnalysis associated with SCORE Project: Imhoff_covid_dPzV_z1k9_analysis.docx.

Pfattheicher

Overarching report as part of the SCORE Project: final report Pfattheicher.docx.

Preregistration and peer review of the replication: Pfattheicher_covid_yZD4-direct replication – edlund – y006.

Abstract for IRB: Abstract score 3.docx.

Consent Form: Consent for score 3-revised.docx.

Ethics form (HRPO): HRPO_Protocol)Submission_form (prefilled)-score 3.docx.

RIT IRB Application: Score 3 irb.docx.

Ethics Form: USAMRMC_Proposal_submission_form (prefilled) – score3.docx.

a priori power analysis: POWER_Pfattheicher _covid_yZD4.zip.

Full materials: Pfattheicher _covid_yzd4_y006_materials.pdf.

Data Dictionary: Data dictionary Pfattheicher.docx.

Data (in SPSS): Pfattheicher _august20,20220_11.04.sav.

Data (in CSV): Pfattheicher_august20,20220_11.04.csvAnalysis associated with SCORE Project: Pfattheicher _covid_yZD4_y006_analysis.docx.

Data type

All files are labeled with descriptive names. Included for both data sets are the preregistrations, the power analysis, the materials, the data dictionary, the data, and the final report generated for the SCORE project.

Format names and versions

Files are .sav, .docx, and .pdf. You would need SPSS, Microsoft Word, and Adobe Reader to open the files in their native format (although there are alternative programs than can successfully open those file types).

Language

American English.

License

Creative Commons Zero (CC0)

Limits to sharing

There are no limits to sharing.

Publication date

June 18, 2022

FAIR data/Codebook

The data codebook should allow any interested party to full understand the files provided. It is believed that this project fully conforms with FAIR guidelines.

Reuse potential

The analyses that were run as part of the SCORE were minimal – a single test of a single hypothesis was run on each data set. However, there are a number of variables that could still be analysed (gender, political orientation, personal knowledge and risk for COVID). Additionally, the number of variables included in both datasets would allow for the combination of both datasets to answer other key questions (as there is no overlap in participants).

Further, the data included in this report represent a snapshot in time (August 20th, 2020) in a continuous evolving pandemic. The data could be combined with other data that exist to allow for a fuller understanding of the evolving psychological response to the COVID crisis.