Bad Science: Quacks, Hacks, and Big Pharma Flacks

We tend to place a lot of trust in the results of medical trials. Why shouldn’t we? As it turns out, some evidence suggests that they might not always be fair tests.

For example, some trials don’t report how they randomize which participants are filtered into the treatment group or the non-treatment group.

In every medical trial there are two groups of patients with a specific disorder: one receives the treatment and the other doesn’t. This approach allows researchers to meaningfully test the effectiveness of the drug.

But not all participants are equal! For example, some participants are known asheartsinks: these patients constantly complain about unspecific symptoms that never improve, and are more likely to drop out or not respond to treatment.

If the next available place in the trial is for the treatment group, the experimenter, wanting a positive outcome for her experiment, might decide that this heart sink shouldn’t participate in the trial – with the result that they test their treatment only on those with a greater chance for recovery.

This has serious consequences: unclear randomization in patient selection can overstate the efficacy of the treatment by 30 percent or more.

For example, one study of homeopathic treatment for muscle soreness in 42 women showed positive results. However, because the study didn’t describe its method of randomization, we can’t be certain that the trial was fair.

Furthermore, patients, doctors or experimenters sometimes know which patient is getting the treatment and which is getting the placebo. In order to be effective, tests need to have something called blinding– i.e., the tester shouldn’t know in which group an individual patient belongs.

Testers can influence their results through conscious or subconscious communication with patient, just as knowledge of which drugs you are taking can influence the way your body responds to treatment.

For example, trials conducted without proper blinding showed acupuncture to be incredibly beneficial, while other tests with proper blinding proved that the benefit of acupuncture was in fact “statistically insignificant.” The difference isn’t trivial!