Marteling van gegevens
Als een wetenschapper heel graag iets wil bewijzen staan hem daarvoor allerhande technieken ter beschikking: hij/zij kan de gegevens zelf selecteren, kan in de analyses factoren die van belang zijn weglaten en nog veel meer.
Deze week verscheen weer zo’n gemanipuleerd onderzoek in de Nederlandse media. Een onderzoeker die graag wilde bewijzen dat dik zijn, een controversieel onderwerp in de VS, je minder lang doet leven, martelde zijn gegevens zo lang totdat het gewenste resultaat er eindelijk uitkwam. Wat hij blijkbaar niet besefte is dat zijn uiteindelijke onderzoeksresultaat bewijst wat de meerderheid van de onderzoeken over dit onderwerp al vond: zwaarlijvige mensen boven de 50 die afvallen verhogen de kans op een vroegtijdige dood.
Een wetenschapper in de VS beschrijft deze week wat er mis is in dit onderzoek.
Confessions extracted under torture are notoriously unreliable. A new study in The New England Journal of Medicine illustrates this point well.
The study analyzes the relationship between weight and mortality risk. In particular, it tries to determine whether being “overweight” (this is currently defined by our public health authorities as 146 to 174 pounds for an average-height woman, and 174 to 208 pounds for an average-height man) is associated with an increased risk of death.
This is an especially controversial issue, for two reasons. First, most Americans whom the government claims weigh too much are in this “overweight” category. Second, many studies find either that there is no increased mortality risk associated with being “overweight,” so-called, or indeed that the risk of death among the “overweight” is actually lower than among so-called “normal weight” individuals.
In particular, a 2005 study led by Centers for Disease Control and Prevention researcher Katherine Flegal found 86,000 excess deaths per year in the United States among “normal weight” people, when comparing their mortality risk to that of so-called “overweight” persons. Because of the current panic over fat, this study caused quite a furor, even though its findings were consistent with many other investigations of the same issue.
It seems the authors of this new study were determined to refute Flegal’s findings – even if they had to subject their data to techniques that violate the scientific equivalent of the Geneva Convention.
The researchers collected data from 527,265 AARP members, who were followed for 10 years. What they found was exactly the same result reported by Flegal and her colleagues: Among both men and women, “overweight” people had the lowest mortality risk. This result, however, was clearly unacceptable. So they began torturing their data.
First, they threw out any subjects who had ever smoked. The justification for doing so in studies exploring the relationship between weight and health is that some people smoke to remain thin, so increased health risk among thin people may be a product of smoking rather than thinness. (In fact in this study, the percentage of “normal weight” people who had never smoked was higher than the percentage of “overweight” and “obese” never-smokers, but never mind).
Yet even after limiting their analysis to never-smokers, the authors found no increased mortality risk among the “overweight” when compared to so-called “normal weight” people. So they then engaged in the methodological equivalent of waterboarding. Bizarrely, rather than using the weights of the subjects at the time they entered the study, the authors asked the subjects what they had weighed at age 50, and used this weight instead. (All the subjects were over 50 at the study’s start, and some were as old as 71; 40 percent did not even respond to the question about what they weighed at age 50, which says something about the reliability of the responses the authors did get.)
This, at last, produced a (modest) increase in mortality risk associated with “overweight,” thus allowing the authors to draw their conclusion that “overweight is associated with an increased risk of death.”
But notice how this result was produced. Since the “overweight” people in the study still had the lowest death risk, even after the authors tossed out 70 percent of their subject pool by limiting their analysis to never-smokers, the study found “overweight” associated with an increased risk of death only among a particular subgroup: people who had been “overweight” at age 50, but were at a “normal weight” when they later entered the study.
In other words, what the study really found is that, for middle-aged “overweight” people, weight loss increases the risk of death significantly! (This, by the way, is a very common finding in studies of this sort).
The authors, needless to say, fail to note this awkward fact, which does not merely contradict, but actually inverts, the public health message their study is intended to bolster. Will journalists covering the study manage to figure this out on their own? Fat chance.
