Get Your Snake Oil Here!

An earnest, ruggedly handsome, middle-aged male looks into the camera.

"My wife takes Dr. Bob's Snake Oil every day.  I haven't been, though.  Just not convinced."

A thoughtful look comes across his face.

"Then I found out about a study on the long-term health benefits of snake oil.

"It turns out Dr. Bob's Snake Oil is what they used in the study.
"I guess my wife was right."

Of course his wife was right.  That goes without saying.  Wives are always right - oh, and yes, the corollary that husbands are always wrong is also true - especially when it comes to matters of health.  That's just the way things are in our society.  Families depend on wives and mothers to make the right decisions about our health.

Remember, "choosy moms choose Jif."  They could never have made that commercial work as well by saying kids like it better.  No, Jif is a peanut butter of clearly superior quality, so it only makes sense that it would be the peanut butter of choice for choosy moms.  If you don't care about what your kids put into their stomachs - which (gasp!) could be some other brand, like Skippy, that just doesn't measure up to the standard of choosy moms - then buy any old kind.  But if you're a choosy mom....

So our friend the commercial actor realizes that his imaginary (or at least off-camera) wife must be right because someone did a study.  He doesn't say who did the study.  Nothing about how it was done, whether it was done in such a way that it could have proved anything at all, or what possible long-term health benefits were being measured.  No, our friend was convinced by the mere fact that someone did a study.

This seems patently ridiculous.  But it is exactly the sort of thing we read and hear in commercial advertisements all the time.  A product is described as being "clinically tested."  That sounds great if you overlook the fact that we're not being told anything about the results of the clinical testing.

Quite often things that have been "clinically tested" are shown to be completely worthless.  Sometimes they are shown to be harmful.  If I told you that getting smacked in the side of the head with a 2x4 has been "clinically tested," would you assume that means it's good for you?  I hope not.  I hope you'd be thinking, yeah, it's been clinically tested and found to be a very effective way to get a concussion - or at least a headache.

Of course some things we believe are good for us have not been proven helpful in clinical trials.  A favorite example used by many of us who are ardent advocates of practicing medicine based on scientific evidence is the parachute. No one has done a study to prove that parachutes are good for you when you jump out of an airplane.

We're doing a study to figure out whether parachutes are good for you.  We are recruiting subjects.  You will be selected at random to jump out of an airplane, either with or without a parachute.  Doing such a study is ethically acceptable only if we really don't know which is better.  It might seem intuitively obvious, but the fact is no one has ever really studied it.  Can we sign you up?

Some studies will not be done, and that's OK.  But if a study is done, that tells you nothing unless you know the particulars of how it was done and what it showed.  It is hard to imagine how our friend in the television commercial for Dr. Bob's Snake Oil could be so clueless.

Well, here's how.  Not only is his wife a long-time, loyal consumer, but the product isn't Dr. Bob's Snake Oil.  It's Centrum Silver, a daily multivitamin "for people over 50."  So the proposition that it is worthwhile seems quite plausible.  Not as intuitively appealing, perhaps, as using a parachute when you go jumping out of a plane, but certainly plausible.

But what does this commercial tell us?  It tells us that someone decided to do a study to determine whether there might be long-term health benefits associated with taking a daily multivitamin.  It tells us nothing about what potential long-term health benefits were considered.  It also tells us nothing about whether the possibility of long-term harm to health was also considered and whether the study was designed to detect any of that.  It tells us nothing about whether any health benefits were actually found - only that someone looked for them.

How many in the television audience decide to investigate such claims for themselves?  I'm guessing the number is small.  And you're guessing the number includes me.  Yep.

It turns out that this was the Harvard Physicians' Health Study II.  This study was actually pretty decent science, at least as far as I have been able to determine. About 15,000 mostly healthy male doctors aged 50 or older were recruited to participate in a decade-long study to see if taking vitamins was beneficial to health.  One arm of the study randomized participants to taking either Centrum Silver or placebo.  At the end of the study, a statistically significant difference was found.  For every 1,000 men in the study, there were 17 new cancers detected among those taking Centrum Silver and 18 among those taking placebo.

Other outcome measures included cardiovascular disease, cataracts, macular degeneration, and "early cognitive decline" (dementia).  No differences were reported for those.  Just as important, the study did not detect any harm caused by taking the multivitamin - because no one looked for harm.

If you have an inclination toward thinking about numbers, you probably already realize that the difference of one cancer per 1,000 participants in the study seems very small.  For those of you who really like math, read the next paragraph for additional insight.  Everyone else skip to the one after that.

We often talk about the risk of something happening in terms of what is called a "hazard ratio."  If the hazard ratio is 1.0 between two groups, then there is no difference, because the likelihood of something happening is exactly the same in one group as the other.  If the hazard ratio is less than one for the treatment group, that's better (or lower risk).  If it's more than one, that's worse (or higher risk).  For any hazard ratio, or any other way of reporting differences, we use what statisticians call confidence intervals.  That's a way of expressing how likely we would be to get the same results if we did the study on a very large population of subjects.  As you can imagine, differences are much more meaningful when they come from bigger studies.  If you flip a coin twice, and you get one head and one tail, that doesn't give you nearly so much confidence that this is a 50-50 proposition as it would if you flipped it ten thousand times and got 5,000 each head and tail results.  So we calculate a 95% confidence interval, and that means if we did this study on the universe of subjects, we could be 95% sure the result would be in this range.  So for this difference between 17 and 18 cancers per thousand subjects, the hazard ratio was 0.92.  The confidence interval was 0.86-0.998.  Notice that the upper end of the 95% confidence interval is so close to 1.0 (no difference) that the calculation had to be carried out to three decimal places to show that it was ever so slightly less than 1.0.

This study was published in the Journal of the American Medical Association (JAMA) last fall.  Pfizer, the maker of Centrum Silver, had its marketing campaign all set to go.  To its credit, Pfizer's television commercial - the one with the earnest fellow admitting his wife was right - says nothing about reducing cancer risk.

I think I know why the one difference that barely - just barely - achieved statistical significance is not being touted by the advertising.  It may be entirely illusory. You see, as it turns out, among the study participants, there were some who had a history of cancer already when they enrolled.  And the very modest reduction in risk of subsequent development of cancer was seen in that group.  If one looks at the others, with no history of cancer, the difference in risk of subsequent development of cancer disappears.

Why wouldn't Pfizer tell us any of that?  Well, if you say taking their vitamin helps just a little, if you've already had cancer, to reduce the risk of developing further cancer, but otherwise not, then you have a much smaller target market.  And the one thing you definitely don't want to say is that this study showed no difference in cancer mortality.

So what is the bottom line on this study?  You knew it when you started reading: snake oil.