In early April, several of my contacts in the gun community sent me a link to an article in Reason (“the nation’s leading libertarian magazine”) and an associated YouTube video.

The authors of the article, statistician Aaron Brown and Reason senior producer Justin Monticello, argued that very few studies of gun control laws test the effects of those laws rigorously, and even those that do have “serious statistical defects.” Consequently, Brown and Monticello conclude that even rigorous empirical studies tell us “nothing” about the efficacy of gun control laws.

What I found interesting was that Brown and Monticello use as a major part of their argument an analysis of gun research by the RAND Corporation, The Science of Gun Policy: A Critical Synthesis of Research Evidence on the Effects of Gun Policies in the United States.

Certainly, I thought, those associated with the RAND study would have something to say about this broadside. When I was at an authors’ workshop on gun violence prevention research recently, I happened to be seated with one of the co-authors of the RAND report, Carole Roan Gresenz. She had not heard of the Reason article/video.

I thought the critique of research underwriting gun violence prevention policies might generate some comment from other workshop attendees, even if only along the lines of “that piece was garbage.” So I emailed links to the article/video to the group. Only one of the attendees responded. Without comment, she shared a link to a Twitter thread written by Andrew Morral that was a direct response to Reason.

Morral is “a senior behavioral scientist at the RAND Corporation, leader of Gun Policy in America, a RAND initiative to understand the effects of gun policies, and director of the National Collaborative on Gun Violence Research, a philanthropy that has awarded more than $21m in research funding to non-RAND researchers.”

As I have not seen Morral’s comment on the Reason article/video outside of Twitter, I unrolled his thread and I am pasting it below.

Begin Morral’s Twitter thread:

This video and accompanying article draw conclusions about the effects of gun control based almost entirely on research I co-led, yet they reached a very different conclusion than we did.  Here I highlight problems that help explain these differences

The article draws 4 conclusions that are not supported by our report. We did NOT conclude that a) all gun research is poor quality, b) the pattern of findings across studies would be expected by chance, c) the field is ideologically biased, or d) gun laws have no effect.

I believe these conclusions are incorrect, and rest on logical, statistical and factual errors.

The article suggests that there are thousands of studies of the effects of gun policy, and RAND concluded that only 0.4% of them provided credible evidence of gun policy effects. Both claims reflect misunderstandings of our report

In fact, of the 21,686 papers we identified in our search, only 357 were RELEVANT to evaluating gun policy, and 123 of those met our inclusion rules. There is not a massive trove of bad research. There is little research on gun policy. More is needed to answer basic questions.

Our report identified methodological weakness in some studies, but the article uses these critiques to make unsupportable generalizations to the whole field. Why would the presence of flaws in a subset of studies lead us to discount evidence from the stronger ones?

Our report found that the literature does not yet support strong conclusions about the effects of many gun policies, typically because no research had been conducted on that policy, or because the data available did not result in statistically significant findings.

The article makes confusing claims about the statistical effects we extracted from the studies. They suggest that since we reported 722 statistical tests from this literature, our finding of evidence for just 18 gun policies is likely the result of chance alone.

As the article seems to concede, this is a misleading claim. Of the 651 unique statistical tests we identified, 163 were significant: many, many more than would be expected by chance if the laws had no effects.

The article suggests our finding that only 1 of many effects favored permissive gun laws suggests that the field is systematically biased in favor of restrictive gun laws. But it isn’t correct that we identified only 1 such effect. There are at least 39 among the 651 we extracted

So we think the article draws a lot of incorrect conclusions. There are few studies of gun law effects, so we need more. Some are flawed, so we should give greater weight to the better ones and seek to strengthen future research. https://www.rand.org/pubs/research_reports/RRA243-1.html

When multiple studies with good methods from different research groups agree on the likely effects of a gun law, dismissing them as chance and bias seems like a biased conclusion.

End of Morral’s Twitter thread.

I am not actively following this “debate,” if there is even one happening. But if you are, please post interesting and appropriate links in the comments so I can keep up.