07 January 2011
Fraud and science
While everyone seems terribly outraged by this, they also seem to be treating the whole thing as some sort of crazy anomaly -- the scientific equivalent of that high school reunion your wife couldn't go to and where you had a bit too much to drink and the (still) hot girl you pined for long ago is now impressed at what you've become and somehow you end up with her panties in your luggage. I mean, why tell your wife? It'll turn both your lives upside down, hurt everyone involved, and it was just a once in a lifetime confluence of alcohol, chance, and reminiscence...
Except that isn't the truth and everyone knows it. While it's not tenable to deploy the mutaween to the thousands of high school reunions in this country, it does seem feasible that we come up with a better accounting system in the world of science (and retractions in particular). At present the editors of surgical journals seem to have no problem telling Retraction Watch that the reason a paper was retracted is, "None of your damn business."
This is why some context is important. The media treating this Wakefield debacle as though it's of singular import casts a shadow that obfuscates the huge number of articles that ought to be retracted. For those too lazy to click that link the authors speculated that between 10,000 and 100,000 unretracted ought to be. (Ah, but maybe their own is among them? Shut up.)
Oh, but this Wakefield thing is just a fluke. Most of those crappy journal articles don't actually hurt people. Right? Right???
Wrong. Let me tell you the story of Werner Bezwoda (as if to emphasize my point, he doesn't even have a Wikipedia page). Back in the 1980s doctors had come up with a new theory for cancer treatment. Well, it wasn't really a new theory, it was just the extension of an old one to an unfathomable degree. The limiting factor on chemotherapy effectiveness at the time was the patient's bone marrow. If you destroyed all of the bone marrow cells (along with the cancer cells) your patient invariably died (although not from cancer). Medicine being what it is, doctors decided, well we need to destroy the bone marrow to kill the cancer, so what we'll do is extract bone marrow from these folks, give them the super megadose (it was actually called that) chemo, and then implant the bone marrow back.
The guys at the NIH who first came up with this idea tried to set up some clinical trials to rigorously test the idea before putting it into practice. Unfortunately they were undermined by a media blitzkrieg that cast them as villains for keeping a potentially lifesaving therapy from dying cancer patients. The public outcry was sufficient that the FDA gave the whole procedure a compassionate use exemption. Public demand for the procedure was fed by our man Bezwoda, a physician claiming extraordinary results with the autologous transplantation after the super megadose chemo protocol he had developed. Clients from around the world were regularly flying down to his clinic at Witwatersrand in Johannesburg, South Africa.
As a metric for how quickly this protocol swept the scientific community, in 1993 alone there were 1,177 papers published on it. In any case, those NIH guys were stubbornly continuing with their trials (try recruiting for a randomized trial when your subjects can jump ship if they don't like their assigned lot).
Then in 1999 Bezwoda opened the annual cancer meeting in Atlanta with a presentation on his results: He found that 8.5 years after megadose chemo and transplantation 60% of his patients were alive, whereas only 20% survived in the control arm. During the afternoon session those guys from the NIH presented their results. Except that their results were, let's say, not good. In one of their studies the researchers found "not even a modest improvement," and complication rates considerably higher than the control arm.
Later in the year a team of researchers pulled together by the president of the American Society of Clinical Oncology flew off to South Africa to take a look at Bezwoda's data. Upon arrival they requested the log books for the 158 patients Bezwoda reported treating. He gave them log books for 58, and said the rest had been lost (oh to live in a world without paperwork retainment requirements). The data he did give them was horrible. One of Bezwoda's purported breast cancer patients was actually a man. The entire thing had been a sham. In essence, Bezwoda's protocol was completely fabricated and his fraud was the sole thing holding up a $4 billion industry that performed the protocol on approximately 40,000 women.
To recap: There was a new experimental procedure for treating breast cancer. The media created such a furor at its restricted use that the government was browbeaten into allowing anyone who could pay (or litigate their insurers into paying, a whole different tangent) have a completely unproven procedure. The procedure turned out to be an epic failure and the exposition thereof went almost completely unnoticed by the mainstream media. (Incidentally, in this case scientists were the ones who exposed the fraud, in the Wakefield case it took a journalist to do so).
I do not have a solution to this madness, but until we, collectively, realize that the status quo is madness we're never going to get any closer to coming up with one.
As a parting note I'm going to quote an email that was sent to Jonah Lehrer by a former academic scientist who now works for a large biotech company:
When I worked in a university lab, we’d find all sorts of ways to get a significant result. We’d adjust the sample size after the fact, perhaps because some of the mice were outliers or maybe they were handled incorrectly, etc. This wasn’t considered misconduct. It was just the way things were done. Of course, once these animals were thrown out [of the data] the effect of the intervention was publishable.
Here we have to be explicit, in advance, of how many mice we are going to use, and what effect we expect to find. We can’t fudge the numbers after the experiment has been done… That’s because companies don’t want to begin an expensive clinical trial based on basic research that is fundamentally flawed or just a product of randomness.Reagan famously said, "Trust but verify." That's not sufficient any more. The new maxim is closer to, "Doubt until you've figured out their incentives to lie."
08 January 2011
Fraud and science, part II
There are reports all over (they're actually just disguised pressers since that's all the journalists read) about a new study in a journal no one reads (Pharmacoepidemiology and Drug Safety; it's the fourth google result when searching its title...). That antipsychotics are being prescribed off label in ever increasing amounts with scant data to support doing so. The actual study (gated, natch) is not linked to by a single one of those articles.
It does not reflect well on the PR acumen of Science (capital S) or scientists that the article was first "published" and the presser released on a Friday. It's as though they're undermining their own cause by ensuring that no one will read this article. (What possible incentive could they have for that? See III.)
Because I'm not actually a journalist, I'm just going to tell you that the presser gives you the highlights fairly accurately and reading the article is (probably) a waste of your time (unlike real journalists, I did link you to the article itself). Here were those highlights:
It does not reflect well on the PR acumen of Science (capital S) or scientists that the article was first "published" and the presser released on a Friday. It's as though they're undermining their own cause by ensuring that no one will read this article. (What possible incentive could they have for that? See III.)
Because I'm not actually a journalist, I'm just going to tell you that the presser gives you the highlights fairly accurately and reading the article is (probably) a waste of your time (unlike real journalists, I did link you to the article itself). Here were those highlights:
- Antipsychotic treatment prescribed during the surveyed doctors' visits nearly tripled from 6.2 million in 1995 to 16.7 million in 2008, the most recent year for which they had data. During this period, prescriptions for first-generation antipsychotics decreased from 5.2 million to 1 million.
- Antipsychotic use for indications that lacked FDA approval by the end of 2008 increased from 4.4 million prescriptions during surveyed doctors' visits in 1995 to 9 million in 2008.
- In 2008, more than half — 54 percent — of the surveyed prescriptions for the new-generation antipsychotics had uncertain evidence.
- An estimated $6 billion was spent in 2008 on off-label use of antipsychotic medication nationwide, of which $5.4 billion was for uses with uncertain evidence.
- Prescriptions for antipsychotics began dropping slightly in 2006, shortly after the FDA issued a warning about their safety.
The difference here is that there are too many genies for us to stuff back into the bottle if and when the whole house of cards falls down. This isn't someone falsifying data and then exhorting their colleagues to follow their invented protocols. This is a bunch of psychiatrists writing prescriptions that may work with no real basis for doing so beyond "everyone else is doing it and it might work." (If this sounds a lot like homeopathy to you, you're not far off... this is Homeopathy 2.0 where we use real drugs because they cost more and require med checks.)
Incidentally, if everyone else is doing it and you aren't, have fun trying to pay your mortgage. When word gets out that you won't exhaust every option for your patients not only will the malpractice suits come raining down, the waiting room will empty out. Patients don't care how well considered your refusal to participate in this charade is, they just want to get better and it's your job to make that happen.
