Medical Studies - What Goes Wrong

Tommy was running wild in his second grade class. His mother Kathy didn't want to put him on Ritalin but she also wanted him to succeed in school. As we talked over his diet, it seemed that Tommy was taking in a pretty decent dose of sugar every day. Kathy said she had trouble getting him to eat anything else, and she had seen a 1994 study in the New England Journal saying that sugar made no difference in a child's behavior. I knew about this study and was able to tell Kathy why I wasn't convinced by the results.

I am glad when my patients have the time and energy to research their own health issues. As I pointed out in our February newsletter, a scientific outlook is part of our human nature. Our curiosity and intelligence serve us well in grasping the complexities of the world in general and healthcare in particular. Understanding how medical studies are constructed gives you a way to evaluate media reports of medical news.

In order to give you more tools to evaluate what you hear, this month's newsletter is about errors that can creep into medical studies. I will list them by type of error and start with the problem that made me doubt the sugar study.

Active Placebo: placebo mimics or duplicates effect of substance under study

The study Kathy referred to looked at about 25 children reported to be sensitive to sugar. They were given mental performance tests and behavior scores before the study and again after three weeks on a high-sugar diet[1] (28 teaspoons a day), three weeks on a high-aspartame diet (which included 6 teaspoons of sugar a day), and three weeks on a high-saccharine diet (which also included 6 teaspoons of sugar a day).

Over the course of this study, scores did not change, prompting the investigators to conclude that sugar did not influence a child's behavior.

This study is one of those in which the placebos (aspartame, saccharine, and six teaspoons of sugar) may have an influence on the study participants. Interestingly, in a postscript to the article the authors had thanked General Mills and the major soft drink manufacturers for their support of the study.

Once she understood the study's flaws, Kathy reengaged herself in the sugar battle and Tommy's teacher stopped complaining.

Let's look at a few other innocent errors that commonly occur.

Selection Bias: study group not representative of patients normally treated

Mary told me that menopause had brought with it sleepless nights and painful intercourse. She was afraid to take estrogen because "studies had shown it was harmful." Mary was referring to the Women's Health Initiative Study published in 2002, which showed that a greater risk of heart disease in women who took estrogen. I pointed out to Mary that the average age of women in the study was 63 and no women with menopausal symptoms were allowed to enter the study. So what the study really showed was that for women who did not need estrogen, estrogen was a net loser.

Mary was 45. We cannot generalize that what is harmful for 63-year-old would be harmful to her. Most physicians today estimate that better sleep and a more comfortable relationship with her husband would far outweigh any risk.

The technical term for this error is selection bias. How you select your test subjects will influence how the results look.

Lead Time Bias: the built in fix

Kim smoked cigarettes and wanted to know if an annual chest x-ray could catch lung cancer early if he were developing it. Yes, I told him, it would. But would it save his life?

Early in the lung cancer wars, some researchers did annual chest X-rays on smokers to try to catch cancer early and save lives. Sure enough, it looked as if they had succeeded. Men who had been coming in coughing and spitting up blood and received the best treatment available at the time died in less than three years. When the researchers started doing the annual X-rays, they caught the cancer before the symptoms began, and the men survived up to four or more years. Success, yes?Not so much.

Here's the catch. Lung cancer takes a couple of years from the time it appears on an X-ray to the time it causes symptoms. (We call this "lead time. ") More careful scientists looked more closely and showed that the screening did not extend life; it just allowed more time for worry. Treatment was equally futile whether started early or late. This type of research error is called "lead time bias."

All studies on cancer screening inherit this bias. This is why the early studies on mammograms showed such a benefit, and why more careful studies done more recently have shown less.

Sometimes screening does enhance survival, but we need to be aware of lead time bias while making that evaluation.

Screening was not going to save Kim's life. A better plan would be to quit smoking.

Generalization Bias: drawing a conclusion from a single study

Recently Ralph asked me about multiple vitamins. He had read in the news that a recent medical study showed them of no benefit.

I told him that there are few medical subjects, no matter how obscure, that haven't had 10 or 20 studies done about them. In no case does every single study show the same result. We go by the preponderance of the evidence. In my monthly reading of the American Journal of Clinical Nutrition, I can think of no issue that hasn't had several articles showing benefit from nutritional interventions, including multiple vitamins.

I told Ralph that we can't just look at a single study hot off the press and ignore the library full of studies that show the opposite. For a more in-depth review of vitamins, I referred him to our newsletter series on vitamins.

Randomization Errors

Even though we assign patients at random, by chance more smokers, diabetics, or other folks with impairments may get into one group compared to the other. Though statisticians will adjust the results to try to level the playing field, errors frequently creep in.

Placebo Easily Differentiated From Test Drug

Sometimes an active drug has side-effects that clearly differentiate it from the placebo. For instance, many psychoactive drugs cause patients to feel drowsy or give them vivid dreams. The knowledge that they are definitely taking an active drug can influence their response, making them subjectively feel better because they expect to.

Sometimes investigators can pick a placebo that has side-effects similar to the active drug to prevent this effect.

What You Can Do

You know that woman in your neighborhood who has a reputation for reliable information? She has that reputation because she checks out what people say or write. She examines new information carefully.

Critical thinking is a discipline that involves a few simple steps:

1. Look at the method – does it seem to you logical that the technique used would produce reliable results?
2. Does it fit in with other things you know? One study that contradicts all previous work is a wake-up call – not to jump in with both feet but to see if another study will get the same results.
3. Bounce the results off a reliable source – not your friend who is always enamored of some new cure or the close-minded old stick-in-the mud, but someone with curiosity and the ability to learn and change.
4. Stay open-minded. If you understand that much of what you know may turn out to be superseded by better knowledge, you are in a position to learn.

Media Unawares

If you follow us on Facebook, you know that journalists often fail to follow the guidelines above.

Next month

Nature does not give up her secrets easily. Now you know some of the errors that can slip into the most carefully designed and executed medical studies. Next month you'll learn about not-so-innocent errors in medical research, errors that threaten your health and raise the cost of your medical care.

Endnotes

[1] A teaspoon of sugar contains about 4 grams of sucrose. The six to ten year old group of children ate an average of 28 teaspoons a day while on the high sugar diet, and a large amount of saccharine or aspartame, along with six teaspoons of sugar a day while on the low sugar diet.

New England Journal of Medicine 1994;330:301-7.