Abstract

Find out how to publish your research as a registered report

Publishing research in peer-reviewed journals typically looks like this: a study is designed, data is collected, the results are (perhaps several years after data collection) written up, and the article is then submitted to a journal. This ‘standard model’ of publishing feels familiar, and it might be hard to imagine alternatives. However, there are (at least) two disadvantages.

First, researchers might invest considerable resources to conduct high-quality studies that are not accepted for publication because of their nil results (see Crow et al., 2023 for an analysis of the (low) prevalence of nil results published in leading criminology and criminal justice journals). Data ends up in the proverbial file drawer and the publication bias in the respective domain is further enhanced.

A second disadvantage of the ‘standard model’ of publishing is that it affords researchers infinite opportunities, or degrees of freedom, to curate the data analysis as well as the presentation of results to attain a compelling narrative. Hypotheses might be revised after key results have been identified (i.e., HARKing); analyses might be repeated with different tests, covariates, or outcomes until significant findings are detected; experimental conditions might be dropped or combined to avoid nil findings (i.e., p-hacking). In the submitted paper, neither of these so-called questionable research practices (QRPs) can be easily spotted. However, QRPs increase the rate of false positives, in other words, the probability that identified effects are not true effects. False positives are unlikely to be replicated and skew the empirical evidence base.

In recent years, a new article format – registered reports – has gained popularity; it promises to address concerns about inflated publication biases and the risk of publishing false positive findings. A key difference between the ‘standard model’ of publishing and the registered report format is that authors receive peer-review feedback before and after data collection (Figure 1).

More precisely, authors submit, initially, a so-called Stage 1 submission. The latter includes three main parts. First, the literature review, study rationale, and hypotheses are presented; these sections resemble those in a ‘standard model’ article. Second, the study design is defined in a Methods section, including, depending on the chosen methodology, the specification of variables and measures, data exclusion criteria, sample characteristics, and/or data collection methods. Third, the planned analysis steps are discussed in detail, referring to data pre-processing and hypotheses tests. The study design and analysis plans are written in future tense. If applicable, completed pilot studies can be presented as well. No data for the main study is or has been collected at this point. Stage 1 submissions may be rejected, accepted, or reviewers may request changes to, for instance, the planned sample size or measures.

Once the Stage 1 submission has been accepted (i.e., in-principle acceptance), authors can collect data in line with the accepted plan. Importantly, if authors do comply with the registered study design and analysis plan, they are guaranteed that their full paper (i.e., Stage 2 submissions) will be accepted for publication, regardless of the results. That is, nil results have the same chance of being published as statistically significant findings. In the Stage 2 submission, the sections that were submitted in Stage 1 are only minimally altered: the Method section will be revised to apply past instead of future tense and the analysis plan may be removed. Results are presented by distinguishing confirmatory and exploratory analyses. The former refer to hypotheses tests and analyses that were confirmed in the Stage 1 submission; the latter include all additional analyses that might seem relevant once confirmatory analyses have been completed.

An overview of journals that offer registered reports is available here. For criminologists, the journals Law and Human Behavior as well as Legal and Criminological Psychology may be of interest. Each journal specifies the precise elements that they expect to be included in a Stage 1 and Stage 2 submission. Typically, ethics approval for the planned study needs to have been granted, and funding to run the study must be confirmed in the Stage 1 submission. Authors are also likely to be asked to indicate in the cover letter a timeline in which they plan to complete the study.