35%, 3%, whatever…
« previous post | next post »
"Straight talk on the FDA’s tumultuous weekend — and new questions about its independence", Stat 8/24/2020:
Matt Herper: So for those just back from a tour of Jupiter’s moons, last night the FDA granted emergency use authorization of convalescent plasma to treat patients with Covid-19. Trump characterized the decision as a major breakthrough. FDA Commissioner Stephen Hahn, who joined him at a news conference to announce the decision, backed him up — but he also misspoke, claiming that giving plasma would help 35 out of 100 people treated.
Adam Feuerstein: Misspoke is being kind. Hahn grossly mischaracterized the benefit of convalescent plasma on Sunday night. I’ll just quote him here: “A 35% improvement in survival is a pretty substantial clinical benefit. What that means is — and if the data continue to pan out — 100 people who are sick with Covid-19, 35 would have been saved because of the administration of plasma.” […]
Matt: That number should be at best 5 out of 100 people. To my eye, it’s more like 3 out of 100 people. And all that is from subgroups of an observational study, so it should be taken with a grain of salt.
Researchers didn’t compare patients who got plasma to a control group. They compared those who got the drug early to those who got it late, and between high levels of antibodies in the plasma and low ones. For the main subset in the study, which was led by the Mayo Clinic, mortality at seven days was 11% for those who got lots of antibodies, versus 14% for those who got few. That’s three out of 100 — again, with a grain of salt.
According to "FDA, under pressure from Trump, authorizes blood plasma as Covid-19 treatment", Stat 8/23/2020:
The documents released by the FDA do not make clear where the 35% figure cited by Trump and Azar comes from. But it may be based on a study conducted by the Mayo Clinic and the NIH, which indicated that plasma treatments appeared to have a small but statistically significant impact on reducing mortality in hospitalized Covid-19 patients who received the infusions within three days of the onset of symptoms, compared with those who got plasma after four days or later.
The death rate after seven days was 8.7% in patients treated early and 11.9% in those not treated until later. But that study, while it included more than 35,000 patients, did not include a placebo group and was not randomized, making it difficult to interpret the data. And it hasn’t been peer-reviewed by other scientists and published in a medical journal.
That would be better than 35% decrease in death rate, 11.9/8.7 = 1.37 — though only about a 3.6% increase in survival rate, 91.3/88.1 = 1.036.
And the two groups compared were different in a bunch of other ways, for reasons that you can probably guess, given the uncontrolled nature of the study. In any case, this is a seven-day survival difference of 11.9-8.7 = 3.2 in 100, not 35 in 100.
The NYT cited a different possible source for the 35% number — Katie Thomas and Sheri Fink, "F.D.A. ‘Grossly Misrepresented’ Blood Plasma Data, Scientists Say", NYT 8/24/2020:
When asked where the 35 percent figure came from, an agency spokeswoman initially directed a reporter to a graph of survival statistics buried in the Trump administration’s application for emergency authorization. The chart, analyzing the same tiny subset of Mayo Clinic study patients, did not include numerical figures, but it appeared to indicate a 30-day survival probability of about 63 percent in patients who received plasma with a low level of antibodies, compared with about 76 percent in those who received a high level of antibodies.
That would be roughly a 20% relative increase in 30-day survival rates, affecting about 13 patients in 100 — so not a good source for a claim of 35%. And again, the two groups were different in lots of other relevant ways.
In the first cited article, Adam Feuerstein opined:
I’ll repeat here what I said on Twitter last night. I’ve spent 20 years listening to — and writing about — biotech CEOs making nonsensical, inflated claims about the benefit of their drugs. It was weird and disconcerting to hear the FDA commissioner do the same.
Hahn could have made a convincing case for granting an EUA to plasma without hyping the data. He’s an oncologist. He knows better.
And Hahn did correct himself on Twitter, after getting viciously dragged in the media:
I have been criticized for remarks I made Sunday night about the benefits of convalescent plasma. The criticism is entirely justified. What I should have said better is that the data show a relative risk reduction not an absolute risk reduction.
— Dr. Stephen M. Hahn (@SteveFDA) August 25, 2020
As others have pointed out, the data doesn't really "show" a relative risk reduction, since incomparable groups were compared.
But the fact that Dr. Hahn is an oncologist is no reason to think that he "knows better", even though he's been associated with many publications where such things matter. As I've observed more than once, medical professionals are often surprisingly ignorant about statistical issues.
Update — more from Stat here.
rpsms said,
August 26, 2020 @ 11:07 am
The most likely way he came up with 35% (without just making it up) is by noticing the two main numbers are "about 12" and "about 8", with a difference of "about 4." 4 is about 1/3 of 12 and so someone might decide to say that "8 is 33% fewer than 12" or more likely simply round it to 35% for no good reason.
This is a massive problem with vague use of language (many people peeve about "{%} fewer" etc. Phrases like "triples the risk" when the risk is very low don't help.
He then went off the rails completely by using this misguided characterization to justify a completely wrong characterization.
mg said,
August 26, 2020 @ 11:26 am
I'm a statistician working closely with medical researchers in clinical trials (starting with the design stage) and sometimes I can't believe how often I have to explain things to MD researchers or correct them, no matter how long they've been working in the field nor how senior they are. Med school doesn't even teach them enough to be able to adequately critically assess journal articles they read.
Tom Dawkes said,
August 26, 2020 @ 11:49 am
This issue was addressed in "More or Less" on BBC Radio 4 this morning at 09:00 (BST) by the presenter Tim Hartford — scourge of dodgy stats. Available to hear at https://www.bbc.co.uk/programmes/m000lzgy
Peter Taylor said,
August 26, 2020 @ 2:43 pm
Not just statistics. In 1993 a medical journal published a paper which is infamous among mathematicians: A Mathematical Model for the Determination of Total Area Under Glucose Tolerance and Other Metabolic Curves (further identifiers elided to protect the guilty). It reports as a novelty a method of numerical quadrature which dates back at least as far as Newton, and which I was taught at age 14 or 15 in secondary school. Not only did it make it through peer review, it has almost 400 citations.
KC said,
August 26, 2020 @ 4:52 pm
Peter Taylor – that's quite funny in a very sad way. Wikipedia says the trapezoid method of finding the area under a curve is much older than Newton even: going back to the ancient Babylonians at least. Recently there's been a lot of unflattering light shed on the things that can pass review and get published in "scholarly journals."
Agree with mg – I'm also a statistician and medical professionals by and large are little or no more educated about math/stats than anyone else for whom it's not their primary job. which is unfortunate in the sense that medical professionals should be making a lot of decisions based on math, or at least published research.
Philip Taylor said,
August 27, 2020 @ 6:54 am
"Harford", not "Hartford", Tom — I only know that because the aristocratic branch of the Harford family contributed to a book written by a recently-deceased friend, Joan Richmond : Nine letters from an Artist: the families of William Gillard.
Andrew Usher said,
August 28, 2020 @ 7:37 pm
Even this article got it wrong in saying that 11.9% to 8.7% was over a 35% decrease – actually it's (11.9-8.7)/11.9 ~ 27%. Handling statistics seems to be hard for most people, but the basic point was preserved at least – the treatment does appear to have a non-negligible benefit (and we understand why it should), so it may as well be used.
Also I don't think it's anything to be ashamed of to rediscover the trapezoidal rule (which must be one of the most rediscovered mathematical methods ever), though surely someone should have told him that it's not really new … I wouldn't say it shows he didn't learn anything in math classes. First, he may have forgotten which happens generally; secondly the way 'numerical quadrature' is or used to be presented in math textbooks gives little clue that it can be applied to 'real world' data and not just nice mathematically evaluable functions. It's quite possible not to connect the two, and certainly the trapezoid rule is the best possible algorithm for the former (and really nowadays, with available computational power, sufficient for the latter, too).
'Harford' and 'Hartford' are surely the same etymologically – can anyone be faulted for not remembering what is essentially a misspelling?
k_over_hbarc at yahoo dot com