notropis

I'm a statistics teacher, with only limited experience conducting surveys, and by no means a statistician, but, in perusing the new Iraqi Death Survey, “Mortality after the 2003 invasion of Iraq: a cross-sectional cluster sample survey” by by Bunham, et al, in the recent Lancet, I've come across several things that disturb me in the reports of data collected, the method of data collection and the method of extrapolation that I'd like to lay out in several posts, here. Then, perhaps, I can entice some people who know more than I do, or just think more clearly than I do, to look at these questions that I have, and address them, either supporting my concerns, or allaying them -- I don't much care which.

I don't really have a horse in this race, after all.

So, to the first, and probably most tenuous concerns, the matter of the believability of several items regarding the data collection.

1. “In 16 (0.9%) dwellings, residents were absent.” When or where can you conduct a survey and find over 99% of the potential respondents at home? Especially in Iraq, where there are, according to NPR, two hundred thousand registered internally displaced persons (link ), and an unknown, but presumably much higher, number of unregistered. So did the surveyors simply skip houses that looked “obviously vacant?” Assuming they did (and this would be a gross breach of statistical process, allowing one’s own biases to influence which houses get sampled, but this happened anyway, see the next post; whether or not they did is hard to tell from the reported procedure -- “Empty houses or those that refused to participate were passed over until 40 households had been interviewed in all locations” -- are the “empty houses” the same as the ones reported above where “residents were absent,” or does “residents were absent” refer to apparently occupied houses where they just didn’t find anyone at home), it still seems amazing to find over 99% of potential respondents at home (and all the more so, since they apparently had to canvass throughout the day – see below.)

(Further puzzling is the statement: “Households where all members were dead or had gone away were reported in only one cluster in Ninewa and these deaths are not included in this report.” Does this mean that in only one cluster were any vacant houses encountered? I can find more than that in upscale suburbs of Minneapolis.)

2. Only “15 (0.8%) households refused to participate.” Now this could be a sign that Iraqis are concerned to get the truth of their plight out, and that’s great. But putting this together with (1) above, we find that in a remarkable 98%+ of the potential households, the head of household or spouse was available and willing to answer the questions (according to the methodology, those were the only ones surveyed.) And this result was achieved, according to the article, on the first pass, without ever re-contacting a household, which the survey teams deemed "too dangerous."

3. In reading the methodology, the impression is given that the surveyors did an incredibly thorough, careful, and considerate job in their work. Yet we read that the teams each consisted of four individuals, who “could typically complete a cluster of 40 households in 1 day.” Now, it’s not clear whether the teams stuck together, or split up into 1s or 2s, but, given time for travel, and assuming 8 hours of surveying time available in a day, if they worked in pairs (which would make the most sense, one male and one female), we find that they spent less than half an hour (24 minutes), on average, per household, yet we’re assured that the following protocols were strictly observed:

“The survey purpose was explained to the head of household or spouse, and oral consent was obtained. Participants were assured that no unique identifiers would be gathered. No incentives were provided. The survey listed current household members by sex, and asked who had lived in this household on January 1, 2002. The interviewers then asked about births, deaths, and in-migration and out-migration, and confirmed that the reported inflow and exit of residents explained the differences in composition between the start and end of the recall period. …. Deaths were recorded only if the decedent had lived in the household continuously for 3 months before the event. Additional probing was done to establish the cause and circumstances of deaths to the extent feasible, taking into account family sensitivities. At the conclusion of household interviews where deaths were reported, surveyors requested to see a copy of any death certificate and its presence was recorded. Where differences between the household account and the cause mentioned on the certificate existed, further discussions were sometimes needed to establish the primary cause of death.”

And further on, we read that official death certificates were produced for 80% of the deaths recorded, all in an average of less than half an hour per interview. I'll have a few more questions about these death certificates in another post.