Review of "An open and reproducible analysis of a controversial criminal law decision: Mission IMM-possible?" as submitted to the International Journal for Crime, Justice and Social Democracy

It’s an important article that I hope to see published, if not in this journal then elsewhere. It provides concrete inspiration for researchers to follow, and, thus, should have an impact. I’m not a legal scholar per se, so will focus my comments on open issues. The goal of the comments is to provide ideas for improving the paper.

Need to be careful about this statement: “to provide, what, according to my knowledge, is the first fully open and reproducible quantitative analysis of judicial decisions” (p. 3). These terms – open and reproducible – are defined differently by people. My suggestion is to just delete the claim to originality. It’s done by doing, not by claiming, after all.

Is this referring to law or legal studies?: “I will now briefly discuss the need for reproducible legal analysis in law, and especially criminal law” (p. 8).

Not to be pedantic, but an “app” is short for “application software.” Fine to refer to apps, but I think the full, formal name should be used first. Also, given the importance of the app to the article, it is probably worth saying a little more about what apps are and their place in research (reproducibility); a sentence or two is all I have in mind.

The author somewhat gets at this (p. 13, p. 28), but there is an argument that for research to be fully reproducible, it can’t rely on paywalled tools like Lexis Advance. Some discussion of this would be interesting, I think. (At the extreme, the argument is you should only cite/references works that are open access, as otherwise people are paywalled from checking your claims and application of those works.)

Not a fan of one sentence paragraphs. See page 15.

Less is more, but an idea (and maybe a bad one) is to say more about how exactly researchers can learn to engage in open science. That assumes readers will be motivated by your article, but unsure what exactly to do next to learn more.