IRRELEVANT EXPERTISE: WHAT HAPPENS WHEN YOU CONSULT A ZOOLOGIST ABOUT STATISTICS

We read:

The British government was advised against publicly criticising a report estimating that 655,000 Iraqis had died due to the war, the BBC has learnt. Iraqi Health Ministry figures put the toll at less than 10% of the total in the survey, published in the Lancet. But the Ministry of Defence's chief scientific adviser said the survey's methods were "close to best practice" and the study design was "robust".

The Lancet medical journal published its peer-reviewed survey last October. It was conducted by the John Hopkins School of Public Health and compared mortality rates before and after the invasion by surveying 47 randomly chosen areas across 16 provinces in Iraq.

Shortly after the publication of the survey in October last year Tony Blair's official spokesperson said the Lancet's figure was not anywhere near accurate. He said the survey had used an extrapolation technique, from a relatively small sample from an area of Iraq that was not representative of the country as a whole.

But a memo by the MoD's Chief Scientific Adviser, Sir Roy Anderson [A zoologist], on 13 October, states: "The study design is robust and employs methods that are regarded as close to "best practice" in this area, given the difficulties of data collection and verification in the present circumstances in Iraq."

To see what statisticians experienced in the research method concerned say, see here and here. I am myself a much published user of that research method and I made the following comments last year:

None of the comments I saw appeared to be by people who are experienced users of cluster sampling -- the method used for the Lancet study. I am a VERY experienced user of cluster sampling -- with many of my academic publications based on it. And the glaring error which rather explains why the study appeared in a medical journal rather than a more statistically sophisticated journal is that there was NO VALIDATION of the survey results. That your survey-takers might just sit down under a tree and "make up" their "interview" results is a routine peril and it is routine to take precautions against it -- usually by going back on a later occasion and checking with the alleged respondents a proportion of all interviews handed in. Just the awareness that a sample of the respondents will be re-interviewed tends to keep the interviewers honest -- though not always so, regrettably. So the results reported in the Lancet study have no credibility at all and must be regarded as garbage.

It is astounding that the authors of the study were so naive. Perhaps they WANTED their interviewers to "fudge" the results -- making clear what the desired results would be, of course.

Another oddity in the Lancet article that suggests something peculiar about the authors is the claim that their interviewers were all DOCTORS -- and not just any doctors but doctors bilingual in Arabic and English. I have never seen the like of that before. Experienced interviewers of some kind were what was needed and that is what is usually used, not doctors. Can we really believe that a whole corps of these rare doctors abandoned their medical duties for so long in order to do something outside their normal expertise? If true it certainly suggests a heavy political committment on the part of the doctors concerned -- exactly what one would NOT want in a study claiming to be objective. To me the whole claim seems like the sort of "gilding the lily" that con-men engage in.

Other critics have noticed other vast implausibilities in the results reported -- the amazingly high (98%) success-rate at getting people to consent to an interview, for instance --- garbage, garbage garbage. And the lie about the death certificates actually shows how bogus the results were.

"Those guys were not even trying to do real research. It was just a propaganda circus. I strongly support Moore's point about the survey's lack of demographic information. That is so unthinkable in survey research that the article would never have been published in an academic journal that knew anything about survey research. The Lancet should stick to medicine.

And as Iraq Body Count note:

"Between January and June 2006, there were 91 violent deaths recorded by the Lancet survey. This would correspond to over 180,000 deaths in the first 6 months of 2006, and an average rate of 1,000 per day. The daily death rate over the same period based on UN reports (which sum Baghdad morgue and Ministry of Health data) is 80 violent deaths per day. Cumulated media reports provide a somewhat lower figure. If the Lancet extrapolation is sound, this would imply a further 920 violent deaths every day (1000 minus 80) which have been recorded by neither officials nor the media. As these are averages, some days would see many more deaths, and others substantially fewer, but in either case, all of them would remain unnoticed."

(For more postings from me, see TONGUE-TIED, EDUCATION WATCH, GREENIE WATCH, POLITICAL CORRECTNESS WATCH, FOOD & HEALTH SKEPTIC, GUN WATCH, SOCIALIZED MEDICINE, AUSTRALIAN POLITICS, DISSECTING LEFTISM, IMMIGRATION WATCH and EYE ON BRITAIN. My Home Pages are here or here or here. Email me (John Ray) here.)

RELIAPUNDIT ADDS: I wiser man told me this:

Some scientists have subsequently challenged the validity of the Lancet study. Questions have been asked about the survey techniques and the possibility of "mainstreet bias".

Dr Michael Spagat of Royal Holloway London University says that most of those questioned lived on main streets which are more likely to suffer from car bombs: "It would appear they were only able to sample a small sliver of the country," he said.

Dr Spagat has previously conducted research with Iraq Body Count, an NGO that counts deaths on the basis of media reports and which has produced estimates far lower than those published in the Lancet.

If the Lancet survey is right, then 2.5% of the Iraqi population - an average of more than 500 people a day - have been killed since the start of the war.

And no one reported it as it happened. This is simply NOT CREDIBLE by sane people. The "anti-war" BDS afflicted Rosie O'donnell-types have no problem believing it though!