Welcome to DU!
The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards.
Join the community:
Create a free account
Support DU (and get rid of ads!):
Become a Star Member
Latest Breaking News
General Discussion
The DU Lounge
All Forums
Issue Forums
Culture Forums
Alliance Forums
Region Forums
Support Forums
Help & Search
119 replies
= new reply since forum marked as read
= new reply since forum marked as read
Is it really a coincidence that people like DrDan, Garen Wintemute, and me, who actually have experience working with the peer reviewed scientific process tend to view things in one way, whereas people like you, and the rest of the marginally educated pro-gunners who get their facts from gun blogs, all swear by NRA talking points? I mean, just a few posts ago, you were claiming to be more qualified than UCDavis to judge academic merit. 
"40% of all sales are at gun shows or private party"?
Using the high figures of this those would only account for 17% of gun sales. The low end is 7%. 
Last edited Sun Aug 12, 2012, 02:52 PM - Edit history (1)
current or former Brady members can chime in and explain this discrepancy. I mean it is right in their lead article on the main page:
The wrong numbers on guns
By: Celinda Lake and Joshua Ulibarri
August 3, 2012 05:04 AM EDT
Following the Aurora, Colo., massacre, many pundits trotted out poll data showing that the American public opposes new gun laws. This finding is disingenuous and if swallowed uncritically, as most of the media has, it can endanger Americans.
***
More recent poll questions indicate that Americans mistakenly believe gun laws are stronger than they actually are. When asked about specifics, most support common-sense gun limits.
First, on behalf of Mayors Against Illegal Guns, Frank Luntz in May polled gun owners and NRA members – those most passionate about gun rights – and found that 57 percent of NRA members believe that “everyone who wants to buy a gun has to pass a background check.”
This is flat-out wrong, since 40 percent of sales are private – conducted at gun shows or over the Internet – and not subject to a federal background check.
By: Celinda Lake and Joshua Ulibarri
August 3, 2012 05:04 AM EDT
Following the Aurora, Colo., massacre, many pundits trotted out poll data showing that the American public opposes new gun laws. This finding is disingenuous and if swallowed uncritically, as most of the media has, it can endanger Americans.
***
More recent poll questions indicate that Americans mistakenly believe gun laws are stronger than they actually are. When asked about specifics, most support common-sense gun limits.
First, on behalf of Mayors Against Illegal Guns, Frank Luntz in May polled gun owners and NRA members – those most passionate about gun rights – and found that 57 percent of NRA members believe that “everyone who wants to buy a gun has to pass a background check.”
This is flat-out wrong, since 40 percent of sales are private – conducted at gun shows or over the Internet – and not subject to a federal background check.
http://dyn.politico.com/printstory.cfm?uuid=336AA1C7-1E2E-4C5D-BC97-C51616819DD6
Brady would not use false statistics in an article that is trying to 'correct' errors in a Gallup poll about American's views on gun control, would they?
Nothing there would stop you from conducting research. Fill out a form to the Joyce Foundation and wait for your check. Before you do that though, you'll need to figure out what constitutes actual research and evidence-based approaches. If you did that you'd probably be disqualified from a Joyce Foundation grant. Stifling indeed. 
those against firearms have a gun-phobia.
What data are you talking about? Link?
I guess you must have been sick the day they explained that a proxy variable doesn't need to be equal, only correlated to the underlying variable, in order to be adequate. Or maybe the peer reviewers once again know a little more than the internet gun fanatics!
Dan, glad to see your work here!
Answered with crickets.
The DU gun-relgionists are sure a different species of "Liberal".
OK, one more time, "self purchasing" is only one of many reasons why the proxy won't be exactly equal to the underlying variable. There are plenty of other sources of error. The correlation is not going to be perfect, and it doesn't have to be. It only has to be significantly correlated, which it is. Its pretty surprising that you are having a hard time understanding this. Oh, and by the way, the results were replicated by Cook and Ludwig using another proxy, fraction of suicides with guns.
Umm, yes there are peer reviewed studies on "both sides". But the (relatively few) "pro-gun" studies come from a small and insular group of ideologues, and the most consequential of these studies -- Kleck's DGU study and Lott's study on CCW -- have been thoroughly refuted by more mainstream researchers. As far as the idea that I don't discuss the studies "on their own", this is laughably false, which is what I've come to expect from you. For example, here is a post I made a while back on the Kleck's study. You should read it, just to compare what it looks like when someone who knows what they are talking about intelligently and thoroughly critiques a study, as opposed to when someone clueless spits out some incoherent and tangential "criticisms" that demonstrate more than anything else that they fundamentally don't understand the statistical methodologies involved.
http://www.democraticunderground.com/discuss/duboard.php?az=show_mesg&forum=118&topic_id=428987&mesg_id=436540
Another long post, but I felt inclined to thoroughly address that moldy lump of toilet paper that is the Kleck-Gertz DGU study.
>What's more, research done by pro-gun criminologists is very frequently of this same variety -- retrospective analysis of demographics/surveys/police databases/etc. Funny that you don't seem to go after the pro-gun research on the same broad methodological grounds.
Very simply, I'm judging the public health research on firearms by the standards to which other medical/public health research is held.
And I propose judging all research by scientific quality rather than applying one standard to the "pro-control" research and another to the "pro-gun" research. If you were at all even-handed about this, you would say that you could not accept Kleck's preposterous DGU numbers until a massive prospective study was done about defensive gun use. It's a true lesson in doublethink to watch you claim that Kellermann's controlling for auxiliary variables "rings hollow", while maintaining the Tooth Fairy levels of credulousness required to accept DGU phone survey results. Yes, case-control studies are not considered as strong as RCTs in the hierarchy of evidence quality. But phone surveys measuring something with a true positive rate of 1% (much less, actually) are not even on the same planet.
Excuse me while I express a derisive snort. Kleck & Gertz issued a persuasive rebuttal to Hemenway's critique. It might also be noted that Hemenway, with his frequent collaborator Deborah Azrael, conducted two surveys on DGUs in 1994 and 1996 (respectively, "Use of Guns in Self-Defense: Results of a National Telephone Survey" and "An Armed Society is a Polite Society? Survey Results," both Harvard University Working Papers and both published in 1996) with very similar methodology to Kleck & Gertz's; that is, prior to the publication of Hemenway's critique in 1997. Call me overly suspicious if you will, but I think there's something a bit... off about somebody claiming that a particular research method is inherently ill-suited to the task only after he's used it twice himself, and had it produce findings that didn't suit his agenda (in the order of 900,000 DGUs annually).
Much the same applies to Cook & Ludwig's study, sponsored by the NIJ and published in 1997. After trying to replicate Kleck & Gertz's experiment, and producing results that, in Cook & Ludwig's own words, were "in the same ballpark that propounded by Kleck and Gertz," C&L then spent the remainder of the paper coming up with ad hoc explanations why their findings were invalid, in broad terms dismissing the methodology as unsuited to the task. But if the method were so obviously unsuited, why declare its unsuitability only after using it? Tellingly, moreover, in the same paper, C&L express total confidence in the veracity of the responses concerning the percentage of respondents reporting one or more firearms being kept in the household. Hence my use of the term "ad hoc": Cook & Ludwig--and Hemenway, for that matter--only express doubts as to the suitability of the survey method only when it produces undesirable results, only as it pertains to those undesirable results, and only after the fact, not before. It's also worth noting that in the intervening 15 years, neither party has produced research on DGUs using a method they did think was suitable.
If you want "play this game" of making comparisons to clear-cut examples of denialism, I can point out that Cook, Ludwig and Hemenway's behavior concerning DGUs is akin to that of creationists who can't actually produce any empirical evidence that supports their claims, and contend themselves with claiming to have poked holes in evolutionary theory, and then committing a false dichotomy that, since the people they oppose are supposedly wrong, they themselves must therefore be right.
(Incidentally I googled the exact titles (quoted) of both Hemenway surveys you give and the only hit I get is your post.)
Again with the insinuations of intellectual dishonesty, but let's take a closer look at the substance. There are extremely valid reasons to question the DGU estimate more than the ownership rate:
1) the DGU estimate is way out of line with many external reality checks
2) unlike the ownership rate, the DGU estimate suffers from extreme sensitivity to false positives, since the true positive rate so low
3) unlike owning a gun, DGU is not a clearly defined act, making for much more potential subjectivity
Interestingly, Kleck uses your same technique in that laughable defense of the DGU survey you link to -- he accuses Hemenway of intellectual dishonesty, but fails to adequately address the substantive criticisms.
I'll discuss those points in more detail, but before doing so, let's first revisit Feynman's Cargo Cult Science lecture, which you brought up in another thread. Good stuff. Particularly the part where he discusses rat-maze experiments. The experiment he highlighted for praise didn't even try to answer any specific questions about rat psychology. Instead, it just found a bunch of different ways that rats "cheat". This sort of research is foundational, because without it, you can't do any sort of valid experiment at all -- if you can't be sure the rats aren't cheating, then any experimental results using that methodology will be questionable.
And this is what is going in here. Kleck's rats are cheating, and Cook and Ludwig explain why. In fact, if you read the actual Cook-Ludwig DGU study (this paper in Journal of Quantitative Criminology), rather than just the report of survey results , then you quickly realize this study is not really about the number of DGUs, it's much more about whether phone surveys are an accurate way of measuring DGUs. In fact, the entire DGU debate is about competing methodologies. It is clear that you get huge numbers with phone surveys and smaller numbers from NCVS -- everyone who has done the requisite calculations has found this. The fact that Hemenway has done a phone survey is no more an indictment of his integrity than the fact that Kleck has examined NCVS data. The substantive question here is which estimates are more believable and why.
And, given that the estimates are drastically different, and that the phone survey numbers are totally out of line with external reality checks, questioning methodology of phone surveys is not denialism but good science, the kind Kleck should have done. If you measure how fast you can ride a bike and come up with 300 +/- 30 miles an hour, it is your duty to ask questions. It doesn't mean you should just ignore the study, though -- on the contrary, the thing to do is examine the methodology, and try to figure out why you got such an absurd number.
Furthermore, the fact that Kleck continues (to this day) to cling to the 2.5M estimate (or in that ballpark) is all the more reason to replicate the study and show exactly where it goes wrong. Cook and Ludwig are not the only ones to do this (e.g. Hemenway and McDowall). Even a cursory glance at these papers shows that the purpose of the studies to examine methodology, rather than just to use phone survey results to estimate DGUs directly (e.g. the McDowall paper is titled "Measuring civilian defensive firearm use: a methodological experiment"
.
False Positives
Perhaps the most important point here -- one that both you and Kleck basically ignore -- is the extreme sensitivity to false positives. Specifically: suppose we call 100 people and ask them about DGU. On average, would even one of those 100, for whatever reason, report a DGU despite not having experienced one in the last 12 months? If the answer is "yes", then Kleck's results belong in the trash. That's how sensitive this survey is to false positives. So when Kleck complains that Hemenway focuses on false positives rather than false negatives, as in:
When H speculates about sources of response error that are plausible, he offers no rationale for why the problems should lead to more false positives than false negatives.
He's totally missing the the point: if the false positive rate is 1% or more, that means almost all of the DGUs are false positives, regardless of the false negative rate. Throughout his rebuttal, you'll find that Kleck spends a lot of time arguing that the false negative rate might be high as well. But this is a losing argument. A high rate of false negatives does nothing to protect the validity of the results -- 1% false positive rate and it's all garbage.
Now, I happen to find a lot of the arguments by Hemenway and others about the potential for FPs in a DGU survey to be quite reasonable. Desirability bias, for example: just browsing through DU Guns, which is one of the tamer gun boards, it is clear that a substantial number of gun owners believe a DGU is a positive, commendable act, and are not shy about sharing their own DGU stories. Indeed, anyone who owns a gun for self-defense must view a DGU somewhat positively, because that's the whole point of owning a gun. Moreover, practically anyone who owns a gun for self-defense will have at least imagined a DGU, and DGU stories circulated among gun owners and non-gun owners alike -- certainly anyone who has a friend with a DGU will know the intricate details of what took place, and thus should they choose to falsely claim a DGU of their own, it would not be hard to fill in some details. These are all characteristics that make FPs more likely -- and remember, we're only talking about 1% of the population, not some massive wave of dishonesty.
But even if you view all of that with skepticism, it must be noted that a FP rate far below 1% is an extraordinary claim, and thus the burden of proof here is much more on Kleck than Hemenway. I say extraordinary because, unlike your tendency to find some pro-control results implausible for no reason other than it doesn't confirm your pre-existing bias, the reason here is not just because 1% is "small": it's that phone surveys generally have FP rates above 1%, particularly when there is social desirability. Hemenway illustrates this point with several examples comparing survey results to known fractions of the population (e.g. 1.5%-7.5% will falsely claim to be dues-paying NRA members, 10% falsely claim to subscribe to Sports Illustrated, 6% claim to have contacted aliens). Oddly, Kleck responds, in section IX, with some examples of his own, of surveys with high false negatives, once again repeating the fallacious argument that potential for false negatives absolves him from the need to bring the false positive rate down. But nowhere does Kleck provide any real evidence that the FP rate is well below 1%, beyond different versions of "we tried really hard".
External Validation
Again, unlike the arguments you make against pro-control studies, the reason that 2 million DGUs is out of the realm of possibility is not just because it seems like a lot to me, it's because those numbers fail multiple concrete external validity checks. When confronted with this fact, Kleck simply engages in speculation and makes increasingly unbelievable excuses, which in most cases are simply transparently incorrect. Specifically:
-- In response to the fact that the DGUs are way out of line with either police reports or NCVS estimates of attempted crimes, Kleck speculates that a large fraction of DGUs may go unreported to police. It's worth quoting this particular whopper:
a large share of the incidents covered by our survey are probably outside the scope of incidents that realistically are likely to be reported either to the NCVS or police
This might sound possible, but for the numbers to work out in any reasonable fashion it would have to be around 90-95% or more DGUs going unreported to police. The thing is, Kleck must take his readers for utter idiots, because his own survey results indicate that the majority of the incidents did get reported to the police:
L. Were Police Informed of Incident or Otherwise Find Out? 64.2
So that pretty much takes care of the "under-the-radar" argument, and I'll let Cook and Ludwig explain some of the ensuing absurdities:
In particular, based on the NSPOF we estimated that there are 265,000 attempted rapes each year in which the victim used a gun and the police were notified; we estimated that an additional 141,000 attempted robberies with gun self-defense were reported to the police. But according to the FBI, the total number of robberies known to the police in 1994 was 619,000, which, together with the NSPOF results, seems to imply that 23% of all robberies known to the police involved the victim defending himself with a gun; and the total number of rapes (including attempts) known to the police in 1994 was 102,000, implying that the number of gun uses against rapists known to the police is 2.6 times the number of rapes or attempted rapes known to the police (Federal Bureau of Investigation, 1995). Another source for estimating the number of robberies and rapes known to the police is the NCVS. According to NCVS estimates from 1994, a total of 719,000 robberies and 137,000 rapes, attempted rapes, or sexual assaults were reported to the police (Bureau of Justice Statistics, 1996). In any case, the comparisons for robberies are implausible, while the comparisons for rapes are impossible.
Kleck also makes the following defense:
H's logic is also fallacious in assuming that one can cast doubt on conclusions based on a large body of data by deriving implausible implications from smaller subsets of the data. Our estimates of total DGUs are likely to be fairly reliable partly because they are based on a very large sample (n=4977), while any estimates one might derive pertaining to one specific crime type are necessarily less reliable because they rely partly on a far smaller subsample, i.e., the 194 reported DGU incidents, of which about 40 were linked to burglaries.
Two things he misses
1) This might be reasonable if only, say, one of the subsets were grossly out of line, but there are many. Also, If Kleck were right, that the inaccuracy were due to variance and small sample size, then some of the individual crimes would be overestimated, and others underestimated. This is what happens, for example, when you break down a national survey into states without enough data -- the state data is unreliable, but for some states the estimate will be too high, and for others too low. But in this survey, the breakdown into individual crimes yields only overestimates: rapes, assaults, robberies, burglaries, all of these numbers come in not just too high, but too high by an order of magnitude. Indeed, simple addition implies that if the estimates for all these individual crimes are too large by a large factor, the total cannot possibly be correct.
2) At the very least, the fact that so many external checks fail by an order of magnitude is a reason to replicate the survey, to see if these are just coincidences due to sample size or if there is really something wrong with the methodology. For example, if other DGU surveys found a total number of around 2.5 million while yielding plausible estimates for these externally verifiable subsets, then perhaps the headline number might be OK. This is exactly what makes other such surveys like Cook and Ludwig's so valuable. It turns out that the failed checks are not just a fluke in Kleck's survey due to sample size, because the same results come up in the other surveys as well. In other words, the fact that phone surveys of DGUs produce absurdly high numbers is a consistently reproducible experimental result, clear evidence that the culprit is actually false positives, rather than variance due to sample size.
-- In response to the fact that his DGU survey would imply that 200,000 people actually shoot a criminal in the course of a DGU annually, about twice the number of people who get hospital treatment for gunshot wounds, Kleck goes on to speculate (again with no evidence) that maybe the vast majority criminals don't go to the hospital. Again, even if you do find this a plausible hypothesis (unlikely, because, as Hemenway points out, surveys of criminals suggest they do seek hospital treatment for GSWs most of the time), a closer look easily shows how absurd the whole thing is. This is because, even though injured criminals may skip the hospital, dead criminals do go to the morgue. And this number of lethal self-defense shootings for that year was about 400, meaning that for Kleck's numbers to add up right, the DGUs must only manage to kill about 0.2% of the criminals they shoot (and hit), again a totally implausible ratio.
Again, Kleck claims that he warned against using the 200K estimate due to accuracy issues, but the fact of the matter is, pretty much nobody believes that significantly less than 10% of DGUs result in someone getting shot (except maybe John Lott...). And therefore, if we accept 2.5M we are forced to accept that the number of defensive shootings in the neighborhood of 200K at least.
-- Kleck dismisses the fact that phone surveys give DGU estimates on the order of 25 times larger than NCVS by repeating that NCVS estimates of DGUs are too low. And this may be the case, but not by a factor of 25. Kleck expects us to believe that 24 out of 25 DGUs would never show up on NCVS just because it doesn't ask about gun use directly. In order to appreciate just how nuts this is, let's actually look at the questions that NCVS survey asks:
41a) Did you do anything with the idea of protecting YOURSELF or your PROPERTY while the incident was going on?
42a) What did you do?
I'll ask you: do you really think that it's remotely feasible that 24 out of 25 people who managed to fend off a robbery with a gun (or a rape, burglary, assault, etc.), would either
1) fail to mention the entire incident to NCVS?
2) respond "no" to 41a?
3) fail to mention something that would qualify as either "attacked offender with a gun" or "threatened offender with a gun" on 42a?
Sure, maybe a few people would, maybe even 50%, though even that's a stretch -- I'm pretty damn sure that if someone tried to rob me and I pulled a gun on them I would say "yes" to 41a and mention the gun in 42a (particularly if I were willing to tell a phone surveyor about it).
It is well known that methodologically, the NCVS is far superior to phone surveys. It will certainly miss some DGUs for reasons such as the fact that it doesn't ask directly about guns. (Although, looking at those questions, it's pretty clear that most reasonable DGU would show up -- if anything, the DGUs missed by NCVS will be from the dubious "other" category). But, if we are talking about legitimate DGUs, where an actual crime was averted, not just brandishing a gun in an escalating argument where the "other guy started it", there's simply no way that NCVS will miss over 95% of the incidents.
Are DGUs really DGUs?
One of the places where the shameless double-standards of the pro-gun crowd are most evident is in their refusal to question whether reported DGUs are true cases in which a G was UD. Indeed, it's pretty funny that you would complain about Kellermann's study not being controlled enough, and yet this particular point does not bother you. Because, quite clearly, the fact that someone claims a DGU does not mean that the use was actually defensive, or even that a "good thing" took place. In fact, it is well known that people tend to portray their own actions in positive light, so asking people whether there own actions were honorable is not exactly a foolproof survey technique.
It is worth mentioning Kleck's response here, just to highlight yet another contradiction:
Regarding possibility (2), we noted that most of the DGUs were linked with the types of crimes (burglaries, robberies, and sexual assaults) where there is little opportunity for participants to be honestly confused about who was the victim and who was the offender.
Hard as it may be to believe, this paragraph appears in the very same document where Kleck warns against relying on the breakdown of overall DGUs into specific crime types, due to sample size. It just so happens that the types of crimes that Kleck highlights here as definitive DGUs are precisely those where the estimates have been shown to be completely out of line with police reports (as well as NCVS) by an order of magnitude. So Kleck wants us to believe that the numbers for individual crimes are accurate enough when rebutting claims of phony DGUs, but not accurate enough to use for external validation. Hmmm...
Another point is that even if legitimate a DGU does occur, this is no guarantee that the presence of a gun improved the situation. It could easily be the case that the gun use was not necessary, even when a survey respondent claims it was -- I'm sure many people use a gun (or anything else) in questionable circumstances and then later insist that their actions were necessary. In fact, it could even occur that a gun used "defensively" actually contributed to escalating a conflict that could have been better resolved by other means.
Now, in principle, the subjective nature of DGUs makes it difficult to determine what fraction of claimed DGUs are legitimate acts of defense that averted a crime or were somehow socially beneficial. And your anti-Kellermann persona here would surely argue to your pro-Kleck persona that claiming DGUs as "averted crimes" without rigorously addressing these issues is an example of, shall we say, "rank dishonesty".
Fortunately, though, Hemenway has actually looked into the question of what fraction of reported DGUs are actually "good". He did a phone survey similar to Klecks, but also recorded descriptions of the alleged DGUs. He then showed the reports to a panel of judges, who found more than half to be likely illegal, and a to a group of criminology students, who found that only about 25% were socially desirable. In other words, even if we were to accept the huge DGU estimates, that does not translate into a net benefit for society.
>What's more, research done by pro-gun criminologists is very frequently of this same variety -- retrospective analysis of demographics/surveys/police databases/etc. Funny that you don't seem to go after the pro-gun research on the same broad methodological grounds.
Very simply, I'm judging the public health research on firearms by the standards to which other medical/public health research is held.
And I propose judging all research by scientific quality rather than applying one standard to the "pro-control" research and another to the "pro-gun" research. If you were at all even-handed about this, you would say that you could not accept Kleck's preposterous DGU numbers until a massive prospective study was done about defensive gun use. It's a true lesson in doublethink to watch you claim that Kellermann's controlling for auxiliary variables "rings hollow", while maintaining the Tooth Fairy levels of credulousness required to accept DGU phone survey results. Yes, case-control studies are not considered as strong as RCTs in the hierarchy of evidence quality. But phone surveys measuring something with a true positive rate of 1% (much less, actually) are not even on the same planet.
Excuse me while I express a derisive snort. Kleck & Gertz issued a persuasive rebuttal to Hemenway's critique. It might also be noted that Hemenway, with his frequent collaborator Deborah Azrael, conducted two surveys on DGUs in 1994 and 1996 (respectively, "Use of Guns in Self-Defense: Results of a National Telephone Survey" and "An Armed Society is a Polite Society? Survey Results," both Harvard University Working Papers and both published in 1996) with very similar methodology to Kleck & Gertz's; that is, prior to the publication of Hemenway's critique in 1997. Call me overly suspicious if you will, but I think there's something a bit... off about somebody claiming that a particular research method is inherently ill-suited to the task only after he's used it twice himself, and had it produce findings that didn't suit his agenda (in the order of 900,000 DGUs annually).
Much the same applies to Cook & Ludwig's study, sponsored by the NIJ and published in 1997. After trying to replicate Kleck & Gertz's experiment, and producing results that, in Cook & Ludwig's own words, were "in the same ballpark that propounded by Kleck and Gertz," C&L then spent the remainder of the paper coming up with ad hoc explanations why their findings were invalid, in broad terms dismissing the methodology as unsuited to the task. But if the method were so obviously unsuited, why declare its unsuitability only after using it? Tellingly, moreover, in the same paper, C&L express total confidence in the veracity of the responses concerning the percentage of respondents reporting one or more firearms being kept in the household. Hence my use of the term "ad hoc": Cook & Ludwig--and Hemenway, for that matter--only express doubts as to the suitability of the survey method only when it produces undesirable results, only as it pertains to those undesirable results, and only after the fact, not before. It's also worth noting that in the intervening 15 years, neither party has produced research on DGUs using a method they did think was suitable.
If you want "play this game" of making comparisons to clear-cut examples of denialism, I can point out that Cook, Ludwig and Hemenway's behavior concerning DGUs is akin to that of creationists who can't actually produce any empirical evidence that supports their claims, and contend themselves with claiming to have poked holes in evolutionary theory, and then committing a false dichotomy that, since the people they oppose are supposedly wrong, they themselves must therefore be right.
(Incidentally I googled the exact titles (quoted) of both Hemenway surveys you give and the only hit I get is your post.)
Again with the insinuations of intellectual dishonesty, but let's take a closer look at the substance. There are extremely valid reasons to question the DGU estimate more than the ownership rate:
1) the DGU estimate is way out of line with many external reality checks
2) unlike the ownership rate, the DGU estimate suffers from extreme sensitivity to false positives, since the true positive rate so low
3) unlike owning a gun, DGU is not a clearly defined act, making for much more potential subjectivity
Interestingly, Kleck uses your same technique in that laughable defense of the DGU survey you link to -- he accuses Hemenway of intellectual dishonesty, but fails to adequately address the substantive criticisms.
I'll discuss those points in more detail, but before doing so, let's first revisit Feynman's Cargo Cult Science lecture, which you brought up in another thread. Good stuff. Particularly the part where he discusses rat-maze experiments. The experiment he highlighted for praise didn't even try to answer any specific questions about rat psychology. Instead, it just found a bunch of different ways that rats "cheat". This sort of research is foundational, because without it, you can't do any sort of valid experiment at all -- if you can't be sure the rats aren't cheating, then any experimental results using that methodology will be questionable.
And this is what is going in here. Kleck's rats are cheating, and Cook and Ludwig explain why. In fact, if you read the actual Cook-Ludwig DGU study (this paper in Journal of Quantitative Criminology), rather than just the report of survey results , then you quickly realize this study is not really about the number of DGUs, it's much more about whether phone surveys are an accurate way of measuring DGUs. In fact, the entire DGU debate is about competing methodologies. It is clear that you get huge numbers with phone surveys and smaller numbers from NCVS -- everyone who has done the requisite calculations has found this. The fact that Hemenway has done a phone survey is no more an indictment of his integrity than the fact that Kleck has examined NCVS data. The substantive question here is which estimates are more believable and why.
And, given that the estimates are drastically different, and that the phone survey numbers are totally out of line with external reality checks, questioning methodology of phone surveys is not denialism but good science, the kind Kleck should have done. If you measure how fast you can ride a bike and come up with 300 +/- 30 miles an hour, it is your duty to ask questions. It doesn't mean you should just ignore the study, though -- on the contrary, the thing to do is examine the methodology, and try to figure out why you got such an absurd number.
Furthermore, the fact that Kleck continues (to this day) to cling to the 2.5M estimate (or in that ballpark) is all the more reason to replicate the study and show exactly where it goes wrong. Cook and Ludwig are not the only ones to do this (e.g. Hemenway and McDowall). Even a cursory glance at these papers shows that the purpose of the studies to examine methodology, rather than just to use phone survey results to estimate DGUs directly (e.g. the McDowall paper is titled "Measuring civilian defensive firearm use: a methodological experiment"
.
False Positives
Perhaps the most important point here -- one that both you and Kleck basically ignore -- is the extreme sensitivity to false positives. Specifically: suppose we call 100 people and ask them about DGU. On average, would even one of those 100, for whatever reason, report a DGU despite not having experienced one in the last 12 months? If the answer is "yes", then Kleck's results belong in the trash. That's how sensitive this survey is to false positives. So when Kleck complains that Hemenway focuses on false positives rather than false negatives, as in:
When H speculates about sources of response error that are plausible, he offers no rationale for why the problems should lead to more false positives than false negatives.
He's totally missing the the point: if the false positive rate is 1% or more, that means almost all of the DGUs are false positives, regardless of the false negative rate. Throughout his rebuttal, you'll find that Kleck spends a lot of time arguing that the false negative rate might be high as well. But this is a losing argument. A high rate of false negatives does nothing to protect the validity of the results -- 1% false positive rate and it's all garbage.
Now, I happen to find a lot of the arguments by Hemenway and others about the potential for FPs in a DGU survey to be quite reasonable. Desirability bias, for example: just browsing through DU Guns, which is one of the tamer gun boards, it is clear that a substantial number of gun owners believe a DGU is a positive, commendable act, and are not shy about sharing their own DGU stories. Indeed, anyone who owns a gun for self-defense must view a DGU somewhat positively, because that's the whole point of owning a gun. Moreover, practically anyone who owns a gun for self-defense will have at least imagined a DGU, and DGU stories circulated among gun owners and non-gun owners alike -- certainly anyone who has a friend with a DGU will know the intricate details of what took place, and thus should they choose to falsely claim a DGU of their own, it would not be hard to fill in some details. These are all characteristics that make FPs more likely -- and remember, we're only talking about 1% of the population, not some massive wave of dishonesty.
But even if you view all of that with skepticism, it must be noted that a FP rate far below 1% is an extraordinary claim, and thus the burden of proof here is much more on Kleck than Hemenway. I say extraordinary because, unlike your tendency to find some pro-control results implausible for no reason other than it doesn't confirm your pre-existing bias, the reason here is not just because 1% is "small": it's that phone surveys generally have FP rates above 1%, particularly when there is social desirability. Hemenway illustrates this point with several examples comparing survey results to known fractions of the population (e.g. 1.5%-7.5% will falsely claim to be dues-paying NRA members, 10% falsely claim to subscribe to Sports Illustrated, 6% claim to have contacted aliens). Oddly, Kleck responds, in section IX, with some examples of his own, of surveys with high false negatives, once again repeating the fallacious argument that potential for false negatives absolves him from the need to bring the false positive rate down. But nowhere does Kleck provide any real evidence that the FP rate is well below 1%, beyond different versions of "we tried really hard".
External Validation
Again, unlike the arguments you make against pro-control studies, the reason that 2 million DGUs is out of the realm of possibility is not just because it seems like a lot to me, it's because those numbers fail multiple concrete external validity checks. When confronted with this fact, Kleck simply engages in speculation and makes increasingly unbelievable excuses, which in most cases are simply transparently incorrect. Specifically:
-- In response to the fact that the DGUs are way out of line with either police reports or NCVS estimates of attempted crimes, Kleck speculates that a large fraction of DGUs may go unreported to police. It's worth quoting this particular whopper:
a large share of the incidents covered by our survey are probably outside the scope of incidents that realistically are likely to be reported either to the NCVS or police
This might sound possible, but for the numbers to work out in any reasonable fashion it would have to be around 90-95% or more DGUs going unreported to police. The thing is, Kleck must take his readers for utter idiots, because his own survey results indicate that the majority of the incidents did get reported to the police:
L. Were Police Informed of Incident or Otherwise Find Out? 64.2
So that pretty much takes care of the "under-the-radar" argument, and I'll let Cook and Ludwig explain some of the ensuing absurdities:
In particular, based on the NSPOF we estimated that there are 265,000 attempted rapes each year in which the victim used a gun and the police were notified; we estimated that an additional 141,000 attempted robberies with gun self-defense were reported to the police. But according to the FBI, the total number of robberies known to the police in 1994 was 619,000, which, together with the NSPOF results, seems to imply that 23% of all robberies known to the police involved the victim defending himself with a gun; and the total number of rapes (including attempts) known to the police in 1994 was 102,000, implying that the number of gun uses against rapists known to the police is 2.6 times the number of rapes or attempted rapes known to the police (Federal Bureau of Investigation, 1995). Another source for estimating the number of robberies and rapes known to the police is the NCVS. According to NCVS estimates from 1994, a total of 719,000 robberies and 137,000 rapes, attempted rapes, or sexual assaults were reported to the police (Bureau of Justice Statistics, 1996). In any case, the comparisons for robberies are implausible, while the comparisons for rapes are impossible.
Kleck also makes the following defense:
H's logic is also fallacious in assuming that one can cast doubt on conclusions based on a large body of data by deriving implausible implications from smaller subsets of the data. Our estimates of total DGUs are likely to be fairly reliable partly because they are based on a very large sample (n=4977), while any estimates one might derive pertaining to one specific crime type are necessarily less reliable because they rely partly on a far smaller subsample, i.e., the 194 reported DGU incidents, of which about 40 were linked to burglaries.
Two things he misses
1) This might be reasonable if only, say, one of the subsets were grossly out of line, but there are many. Also, If Kleck were right, that the inaccuracy were due to variance and small sample size, then some of the individual crimes would be overestimated, and others underestimated. This is what happens, for example, when you break down a national survey into states without enough data -- the state data is unreliable, but for some states the estimate will be too high, and for others too low. But in this survey, the breakdown into individual crimes yields only overestimates: rapes, assaults, robberies, burglaries, all of these numbers come in not just too high, but too high by an order of magnitude. Indeed, simple addition implies that if the estimates for all these individual crimes are too large by a large factor, the total cannot possibly be correct.
2) At the very least, the fact that so many external checks fail by an order of magnitude is a reason to replicate the survey, to see if these are just coincidences due to sample size or if there is really something wrong with the methodology. For example, if other DGU surveys found a total number of around 2.5 million while yielding plausible estimates for these externally verifiable subsets, then perhaps the headline number might be OK. This is exactly what makes other such surveys like Cook and Ludwig's so valuable. It turns out that the failed checks are not just a fluke in Kleck's survey due to sample size, because the same results come up in the other surveys as well. In other words, the fact that phone surveys of DGUs produce absurdly high numbers is a consistently reproducible experimental result, clear evidence that the culprit is actually false positives, rather than variance due to sample size.
-- In response to the fact that his DGU survey would imply that 200,000 people actually shoot a criminal in the course of a DGU annually, about twice the number of people who get hospital treatment for gunshot wounds, Kleck goes on to speculate (again with no evidence) that maybe the vast majority criminals don't go to the hospital. Again, even if you do find this a plausible hypothesis (unlikely, because, as Hemenway points out, surveys of criminals suggest they do seek hospital treatment for GSWs most of the time), a closer look easily shows how absurd the whole thing is. This is because, even though injured criminals may skip the hospital, dead criminals do go to the morgue. And this number of lethal self-defense shootings for that year was about 400, meaning that for Kleck's numbers to add up right, the DGUs must only manage to kill about 0.2% of the criminals they shoot (and hit), again a totally implausible ratio.
Again, Kleck claims that he warned against using the 200K estimate due to accuracy issues, but the fact of the matter is, pretty much nobody believes that significantly less than 10% of DGUs result in someone getting shot (except maybe John Lott...). And therefore, if we accept 2.5M we are forced to accept that the number of defensive shootings in the neighborhood of 200K at least.
-- Kleck dismisses the fact that phone surveys give DGU estimates on the order of 25 times larger than NCVS by repeating that NCVS estimates of DGUs are too low. And this may be the case, but not by a factor of 25. Kleck expects us to believe that 24 out of 25 DGUs would never show up on NCVS just because it doesn't ask about gun use directly. In order to appreciate just how nuts this is, let's actually look at the questions that NCVS survey asks:
41a) Did you do anything with the idea of protecting YOURSELF or your PROPERTY while the incident was going on?
42a) What did you do?
I'll ask you: do you really think that it's remotely feasible that 24 out of 25 people who managed to fend off a robbery with a gun (or a rape, burglary, assault, etc.), would either
1) fail to mention the entire incident to NCVS?
2) respond "no" to 41a?
3) fail to mention something that would qualify as either "attacked offender with a gun" or "threatened offender with a gun" on 42a?
Sure, maybe a few people would, maybe even 50%, though even that's a stretch -- I'm pretty damn sure that if someone tried to rob me and I pulled a gun on them I would say "yes" to 41a and mention the gun in 42a (particularly if I were willing to tell a phone surveyor about it).
It is well known that methodologically, the NCVS is far superior to phone surveys. It will certainly miss some DGUs for reasons such as the fact that it doesn't ask directly about guns. (Although, looking at those questions, it's pretty clear that most reasonable DGU would show up -- if anything, the DGUs missed by NCVS will be from the dubious "other" category). But, if we are talking about legitimate DGUs, where an actual crime was averted, not just brandishing a gun in an escalating argument where the "other guy started it", there's simply no way that NCVS will miss over 95% of the incidents.
Are DGUs really DGUs?
One of the places where the shameless double-standards of the pro-gun crowd are most evident is in their refusal to question whether reported DGUs are true cases in which a G was UD. Indeed, it's pretty funny that you would complain about Kellermann's study not being controlled enough, and yet this particular point does not bother you. Because, quite clearly, the fact that someone claims a DGU does not mean that the use was actually defensive, or even that a "good thing" took place. In fact, it is well known that people tend to portray their own actions in positive light, so asking people whether there own actions were honorable is not exactly a foolproof survey technique.
It is worth mentioning Kleck's response here, just to highlight yet another contradiction:
Regarding possibility (2), we noted that most of the DGUs were linked with the types of crimes (burglaries, robberies, and sexual assaults) where there is little opportunity for participants to be honestly confused about who was the victim and who was the offender.
Hard as it may be to believe, this paragraph appears in the very same document where Kleck warns against relying on the breakdown of overall DGUs into specific crime types, due to sample size. It just so happens that the types of crimes that Kleck highlights here as definitive DGUs are precisely those where the estimates have been shown to be completely out of line with police reports (as well as NCVS) by an order of magnitude. So Kleck wants us to believe that the numbers for individual crimes are accurate enough when rebutting claims of phony DGUs, but not accurate enough to use for external validation. Hmmm...
Another point is that even if legitimate a DGU does occur, this is no guarantee that the presence of a gun improved the situation. It could easily be the case that the gun use was not necessary, even when a survey respondent claims it was -- I'm sure many people use a gun (or anything else) in questionable circumstances and then later insist that their actions were necessary. In fact, it could even occur that a gun used "defensively" actually contributed to escalating a conflict that could have been better resolved by other means.
Now, in principle, the subjective nature of DGUs makes it difficult to determine what fraction of claimed DGUs are legitimate acts of defense that averted a crime or were somehow socially beneficial. And your anti-Kellermann persona here would surely argue to your pro-Kleck persona that claiming DGUs as "averted crimes" without rigorously addressing these issues is an example of, shall we say, "rank dishonesty".
Fortunately, though, Hemenway has actually looked into the question of what fraction of reported DGUs are actually "good". He did a phone survey similar to Klecks, but also recorded descriptions of the alleged DGUs. He then showed the reports to a panel of judges, who found more than half to be likely illegal, and a to a group of criminology students, who found that only about 25% were socially desirable. In other words, even if we were to accept the huge DGU estimates, that does not translate into a net benefit for society.
