notropis

I find some major reasons to be skeptical of the data collected, and the method of collection.

The first three are questions that lead me to doubt the credibility of the survey teams, in spite of the authors’ assurances:

1) 99% of residences had people at home?
2) 98% of residences had heads of household (or spouses) at home and willing and able to provide the requested information?
3) Thorough explanations and double-checking and proofs of information were possible in well under ½ hour per interview?

The next three lead me to question the inherent bias allowed in the data collection:

4) Samples of size 1 - 3 (the number of clusters chosen from all but two of the governorates) chosen “proportionally to population size” can’t possibly reflect the actual urban-rural makeup of an individual governorate, and given the small number of governorates, won’t reflect the makeup, even taken in aggregate. You can "bootstrap" all you want, but you can't bootstrap with a sample size of 1 (and yes, one cluster is, in many respects just "1.") Moreover, no amount of bootstraping will make a demographic that's completely absent (like "rural Iraqis in X province") suddenly appear.

5) Persons living on non-officially-recognized and named streets were systematically ignored, biasing considerably against those living in urban slums, internally-displaced refugees, and rural Iraqis, and this influence was completely overlooked.

6) The field manager could select sampling sites at whim, and these could be changed at the “responsibility and authority” of the interview team.

I'm sorry, but, even with the amazingly advanced statistical analysis done on the numbers, these fundamental questions about the worth of the data collected undercut any validity of the survey. "Garbage in = garbage out" remains as true as ever, as does "data of questionable validity in = conclusions of questionable validity out" (although that's not nearly so catchy.)

# posted by Notropis @ 8:22 PM