IMO ...destroying the data that establishes the validity of how a study was conducted...
does more than cast doubt, it turns science (however imprecise) into an exercise in pure faith. It's just never done unless people are supremely incompetent or dishonest. Some heads needs to be mounted on pikes outside the gate -- either the reporter's and editor's or the scientist's.
As for bias in the study, I'm sure there are some operating. It would be impossible to do one under those conditions without some. However, it's not clear that shootings would be more frequent on busy streets. If I were going to off someone I'd look for a place where I'd be less likely to encounter a platoon of Marines.
Regardless, the difference in the numbers is so great and the methodology reported sufficiently reasonable that it's up to the doubters to replicate it if they want to dispute it. Chanting "Nah, Nah, Nah, I don't/won't hear you" won't suffice.
I Just Don't Get Used To People Lying
Hm, and you're in the legal profession? Cognitive dissonance alert.
And you got Ezra catching Kudlow in a series of lies, and Ezra catching Bush with the "I never said 'stay the course'". A blog I visit the other day had Lomborg next to Diamond;global warming no big deal. Henry at Crooked Timber is just shocked at a NYT book review of Jacob Hacker.
Saisegly has a short philosophical post on the "value of truth." I think. Wilkinson comments well there.
They lie. There are millions of "them" willing to lie and willing to cover each other's lies. I just think a "liberal" society requires a minimum level of consensus on methods of justification (Rawls?), and if that consensus is lacking, the liberal elements are just gonna lose. Being all righteous just don't make up for a half million, a million before its over, dead Iraqis.
Lancet could have said 2 million, asked for evidence, said "Fuck You." I would have backed them. Assume more subtlety, but my point should be clear. The peer review didn't and won't help at all.
IANAL but it seems like the legal profession, as opposed to the political one, is one where there is a real and appreciable difference between truth and lies. Just being able to make a persuasive argument for either side of a case doesn't necessarily imply dishonesty; perhaps it implies a willingness to ignore the relative strength of arguments. But anyway, it's actually illegal to lie in a courtroom, something which can be said for very few other places in America.
Reading this Geoff Eley book on the history of SDP parties in Europe. I ain't no anarchist this week.
The Berlin Workers newspaper had 50k readers before WWI. Most of those got their socialism from Kautsky. The workers didn't know from labor theory of value vs marginalism or rational choice or whatever. The workers/voters understood a few, probably empirically shaky, concepts. Did it matter? Should scientific rigour have mattered to the writers and theoreticians of the Left?
If labor theory of value gets us Universal Health Care, even if it's bullshit, should we who study care?
How the fuck did science sneak its way into politics?
2: This isn't a guarantee, but after seven years of working as a lawyer, I'd say that lawyers are, as neil suggests, less likely to actually lie to you than most other people. They're probably as or more likely to leave you believing something that is untrue by carefully choosing what truths they are going to tell you and what they are going to withhold, but genuine lying has a real tendency to get you in concrete trouble when you're practicing law, and so lawyers mostly don't.
That's pretty much what throws me -- I'll be as suspicious as anyone about hidden information, concealed but necessary context, or whatever, but I have a hell of a time allowing for the possibility that a direct statement of facts known first-hand to the speaker is going to be false.
"Either Lowenstein has failed to grasp the simple, basic core of Hacker’s argument or, for whatever reason, he has decided to give a misleading impression of what the book actually says. This isn’t the first highly peculiar review of the book - see Kevin Drum on Brink Lindsay’s review which similarly fails more or less completely to address Hacker’s actual argument. Something funny is happening here." ...Henry at CT
Yeah, Henry it really is funny.
DeLong has been desperately defending Greg Mankiw, who teaches at Harvard or somewhere, for years now. The economic blogs are hilarious, with their polite assumptions of good faith.
The full details from Kaplan at Slate:
"The article in Science reports that Gilbert Burnham, the chief Hopkins researcher, disputes the Oxford-Holloway team's premise. He says some side streets "were included" in the sample and that "the methods section of the published paper is oversimplified." When the Science reporter asked Burnham to specify how many such streets were included, Burnham replied "that he does not know exactly how the Iraqi team conducted its survey; the details about neighborhoods surveyed were destroyed 'in case they fell into the wrong hands and could increase the risks to residents.' "
I sent Burnham an e-mail asking him about this point. He replied:
I did not ever tell the writer from Science that the raw data have been destroyed. Absolutely NOT! It is sitting right here! What I did say is that our Iraqi colleagues are very concerned about security, not just theirs but the neighborhoods they surveyed. They have asked us for the moment not to release the data to others as there might be some identifiers there. I am sure that we can remove any unique identifiers, but I am bound to honor their requests, as they have staked so much in collecting the data. We will be discussing this over time with our Iraqi colleagues, and I would imagine that in due course we can make it available to those interested.
I wrote back, saying I was puzzled. "Identifiers" are often coded in survey data so identities can be kept secret. Didn't anybody do that here? He replied:
Under human subjects regulations we could not keep unique identifiers, so we limited the information collected—such as street and house numbers. The team did not write down information on the forms on the specific decision making process for each location. (Italics added.)
If I understand this statement correctly, I'm astonished. It sounds as if he's saying he didn't destroy the data because they never existed in the first place. If that's the case, how does Burnham know whether his instructions on methodology were followed at all? How can anyone verify the findings? And this is a peer-reviewed article. Who were these peers? And what did they review?
Burnham did not write back to my subsequent requests. The other main author, Les Roberts, did not respond to any of my e-mails."
Yeah, I read that throught the link from Lambert.
This: It sounds as if he's saying he didn't destroy the data because they never existed in the first place.
is bizarre. The relevant data is the addresses of the clusters, and I understand Burnham to be saying that they have that information.
This doesn't sound like out-and-out lying to me. "Data" is an imprecise enough term that it's not clear to me what particular data exists and what does not.
Kaplan is a bit of a kook, but his work gets published in real news outlets, so I don't think it's OK to completely blow him off. And Kaplan says he got in an email from the author, saying "Under human subjects regulations we could not keep unique identifiers, so we limited the information collected—such as street and house numbers. The team did not write down information on the forms on the specific decision making process for each location." That sounds weasely to me.
Of course, the whole "I have never said stay-the-course" stuff is bullshit of the highest order, even worse than "I was for it before I was against it", and I hope that whomever is running Democratic campaigns makes good use of it. Not admitting mistakes is one thing, not believing that you made mistakes is something much more serious.
"Destroyed" in the face of a denial from Burnham that anything was destroyed, looks like lying or astonishing sloppiness to me on Bohannon's part. Kaplan was relying on Bohannon, as far as I can tell.
I think it is completely within the norms of good science to not record certain kinds of identifying information--like street names and house numbers. This sort of thing happens all the time. Identifying information is removed from tissue samples, for instance.
Protecting the confidentiality of subjects is a legitimate research goal. There is nothing unscientific about it.
They didn't include the exact identities of all the houses the people went to, but why would that be needed? Surely anybody who'd be devoting the legwork to verifying this study wouldn't go back to the same houses just to see if they really said what Burnham's surveyors said they said.
I have the impression that the addresses were not kept. The raw data are the answers to the specific questions asked. Addresses are part of the research methodology.
10:"not believing that you made mistakes is something much more serious."
Bush knows what he said. Bush knows he'll get caught. Bush knows it doesn't matter. I just quoted in this comment an example as to why it doesn't matter.
Gonna go read some Schmitt on why liberal societies fail. Sometimes I don't understand what's going on.
You know, on re-reading what Burnham said I'm not sure what data he's saying were collected. Regardless, it's very different from saying that the data were destroyed.
9 - The researchers know the general location of the clusters, yes. But the methodology had them dropping teams down in the center of the cluster, and then wandering around until they found someone to talk to that met the right criteria. The address of such a person--including, I imagine, whether they lived on a side street or a main street--does not seem to have been recorded based on these quotations.
But what is the problem? The researchers have a legitimate interest in protecting the identify of their subjects. Look, people fill out surveys all the time where there is no record on the survey of who filled it out. This is legitimate.
"Kaplan is a bit of a kook, but his work gets published in real news outlets, so I don't think it's OK to completely blow him off."
Did you just fall off the turnip truck, boy?
17: Yes, if they noted the general area it's fine. Noting the exact addresses could well be creating more death certificates.
What I haven't seen in the objections is any evidence for a "Main Street" effect. It makes sense that it exists when an armored column advances but not necessarly when an insurgency looks for targets of opportunity.
The 'Main Street' effect is based on a misunderstanding of the survey methodology. The survey methodology actually excludes main streets from consideration.
The third stage consisted of random selection of a main street within the administrative unit from a list of all main streets. A residential street was then randomly selected from a list of residential streets crossing the main street. On the residential street, houses were numbered and a start household was randomly selected. From this start household, the team proceeded to the adjacent residence until 40 households were surveyed.
In other words, streets which were surveyed were explicitly not main streets -- they were streets that crossed main streets at some point. The survey was begun at a random location on that street, not at the main intersection.
The objection was bunk anyway since the study was not meant to analyze how many people died in their homes.
Did you just fall off the turnip truck, boy?
If you publish a study, and someone who gets articles published in reputable news sources asks you questions and you fail to answer, this will get reported, and make people ask questions. It may behoove said authors to reply to questions that might be being asked in bad faith in spite of the annoyance. You don't need to be an analytic philosopher to figure that out.
In this day and age, blowing off people who work for "real news outlets" is wise policy, because the "real news outlets" have incredibly low reputations among those people who know how to read. Especially Slate, many of whose writers work to a kneejerk contrarian template that's 10-15 years out of date.
This whole debate is bogus, because the main-street effect, if it's even real at all, would have only a moderate trimming effect on the numbers. The first round of attacks on the study pretty much failed, so now we have a second round of bold contrarians defying the conventional wisdom in a slightly different way than the rest of their herd defied the conventional wisdom.
The Lancet report is the first serious attempt to figure out how many Iraqis have died, and if it's off by a factor of 2 or 3 the reality is still far, far worse than the liars in the White House are saying it is. Everyhting we know from other nonstatistical sources is consistent with the general idea that many more Iraqis have died that the administration is willing to admit, and everything we know from other sources tells us that the Bush administration is systematically blatantly dishonest.
But the argument now is about methodology, and not about what happened in Iraq, so Bush won that little skirmish.
I think that a $50,000 survey would be a minimal investment for a good-sized US newspaper. I'll subscribe to whichever of them dares to try to reproduce the Lancet study's results.
No shit the Bush administration is clearly dishonest and Iraq is a gigantic clusterfuck the scale of which has thankfully not been seen in recent history. I mean, even the Vietnam war had a certain internally consistent logic to it, which as near as I can tell is completely lacking in this current disaster. Perhaps the Soviet invasion of Afghanistan in comparable, but Afghanistan was already a bit of a third world hellhole, and the Soviets screwing things up was not a huge surprise.
You don't need a scientific paper to tell you that; search Google News for "iraq killed attack -soldier" and you'll see that things are fucked up over there. But a paper as politically charged as this was clearly going to come under a withering storm of criticism; a bit of effort to anticipate the likely avenues of attack and have prepared, clear responses isn't too much to ask.
I personally believe that they need not have tried to be so precise, and should have just given very rough figures. The statistical precision is only intelligible to a small number of trained people, and the political argument was carried out between statistical ignoramuses. Furthermore, a few statistically sophisticated people were sure to step forward to trash the report, either from professional purism or for ideological motives.
The fact that this is the first careful study of any type makes the trashers look especially bad. And as d^2 has said (and Jackmormon here), skeptics could arrange to have another study done if they really cared. But the US government and the prowar faction simply do not want the deaths to be counted.
Isn't this the second careful study, after the one that was published in the Lancet almost exactly two years ago?
Regardless, you are exactly on target on the wisdom of reaching for a high level of statistical precision. Going for it wasn't necessarily a bad idea, but going for it and missing (if it turns out that's what happened) was.
Unlike John Cole, I didn't vote for these bastards, but I'm still ashamed that I thought that the war might have been a good idea. I can only plead living in San Franciso and being accustomed to what passes for political discourse here as a defense.
I don't actually think that they missed. I suspect that they did a good job both times, but that in the nature of the case someone would be sure to quibble the results into uselessness.
As Leibniz said somewhere, people would argue about the multiplication table if there were enough money in it.
4/6: really? I was asked by a senior associate to lie to opposing counsel about the date on which we became aware of the existence of a certain document on my very first assignment on joining a firm out of law school. What I remember most vividly was the fact that there was absolutely no good reason to lie in that particular cicurmstance; the request seemed almost instinctual, or perhaps habitual. (I personally believe there are few if any circumstances in which there is good reason to lie, but what I mean is that there were no grave consequences that could have possibly resulted either for us or for our client for telling the truth. The worst thing that could have happened would have been a very minor loss of face on our side, but it would be hard to overstate how inconsequential this would have been in context. In context to an outright lie, which if outed would obviously create all sorts of very bad complications.)
For the record, I ignored his request to lie and instead buried this bit of information in the sort of deliberately-deceptive-but-not-technically-false cloak LB describes in 6, and the details were never pressed by the other side so it never became an issue. (The more complicated question of why this sort of soft deception is morally superior to an outright lie I'll defer for another discussion.)
30: Huh. IME that sort of thing happens rarely, and it tends to come from a few erratic people (that is, I've worked with people who'd do that, but they were recognized as weirdos). Much more common is "Dammit, what can we say that's literally true but doesn't give them the answer they want?"
(The more complicated question of why this sort of soft deception is morally superior to an outright lie I'll defer for another discussion.)
I don't think it is, but it is easier to deal with.
My HS friend who practiced law for a few years said he finally got sick of it when he realized that his client was lying to him, the opposing client was lying to his own attorney, the opposing attorney was lying in court, and that my friend himself had been put in the position of saying things that he knew were not true, though no one could not prove that he knew they were not true.
He did bottom-of-the-barrel criminal and civil law in a poverty-stricken rural area.
In retrospect I think he *was* recognized as a weirdo. But I think there are an uncommonly large number of such weirdos in the legal profession. There're still a minority, but they're not an insignificant one.
A DOLPHIN!!! I SAW A DOLPHIN!!!
33: Oh, they're certainly out there, but they aren't the norm. Say, less than one out of five? Which is, given my impression of the rest of the world, a pretty good average for the legal profession.
Clients and witnesses lie like rugs. The hardest part of preparing a witness for a deposition is getting across that you really don't want them to lie. I've actually had to say things like: "If you're understanding something I say to be a hint to say something untrue, nudgenudge winkwink, you're mistaken. Everything you say should be absolutely literally true, otherwise it'll just come back to bite us."
though no one could not prove that he knew they were not true
Geez John, there's no need to be so negative.
The dolphin has scary-looking teeth.
I was a juror in a hllbillyish neighborhood dispute. The defendant and victim had switched roles from a previous trial. The defendant didn't testify, and I was pretty sure that 6 of the 7 witnesses were lying, though perhaps they just seemed shifty all the time.
The climax of the case came whan the victim's wife accused the defendant of stealing her underwear off the clothesline. I bit my tongue and buried my face because I didn't want to cause a mistrial.
The unsuccessful prosecutor was a hip-looking, nice-looking young woman. I tried to imagine what she'd say about the case at the bar after work.
37 -- I'm telling you, it's obvious what happened to the kitty.
29, man I gotta figure out where he said that, or who actually said that. That is too cool.
30: It isn't morally superior; I am perfectly willing to believe it is more effective /safer in law.