COVID-19 testing: a warning
« previous post | next post »
Everyone is talking about the importance of more extensive COVID-19 testing in determining who is infected, and (eventually) who has been infected.
But nearly all the discussion that I've heard and read has been based on the assumption that the relevant tests are accurate. And this assumption is false — the available tests for this condition seem to be even less accurate than medical tests generally are. Thus Saurabh Jha, "False Negative: COVID-19 Testing's Catch-22", Medpage Today 3/31/2020:
In a physician WhatsApp group, a doctor posted he had a fever of 101 degrees Fahrenheit and muscle ache, gently confessing that it felt like his typical "man flu" which heals with rest and scotch. He worried that he had coronavirus. When the reverse transcription-polymerase chain reaction (RT-PCR) for the virus on his nasal swab came back negative, he jubilantly announced his relief.
Like Twitter, in WhatsApp, emotions quickly outstrip facts. After he received a flurry of cheerful emojis, I ruined the party, advising that despite the negative test, he assumes he's infected and quarantine for two weeks, with a bottle of scotch.
It's believed that the secret sauce to fighting the pandemic is testing for the virus. The depth of the response will be different if 25% of the population is infected than 1%. Testing is the third way, rejecting the choice between death and economic depression. Without testing, strategy is faith-based. But what'll you do differently if the test is negative?
That depends on the test's performance and the consequences of being wrong. Though coronavirus damages the lungs with reckless abandon, it's oddly a shy virus. The Chinese ophthalmologist who originally sounded the alarm about coronavirus, Li Wenliang, had several negative tests. He died from the infection.
In one study, RT-PCR's sensitivity – that's the percentage of infected testing positive – was 70%. Of 1,000 with coronavirus, 700 test positive but 300 test negative.
I haven't been able to find a clear estimate of the false positive rates for the standard RT-PCR kits — or any of the relevant statistics for the numerous other test kits that are being rolled out, or for the antibody tests being developed and tested. But everyone should understand what the issues are. The following explanation comes from "Biomedical nerdview", 9/26/2014:
We start with a 2-by-2 "contingency table", relating test predictions and true states or outcomes:
| Reality is Positive (P) | Reality is Negative (N) | |
| Test is Positive | True Positive (TP) | False Positive (FP) |
| Test is Negative | False Negative (FN) | True Negative (TN) |
In the context, the "sensitivity" is the true positive rate: TP/P, the proportion of real positives that test positive.
The "specificity" is the true negative rate: TN/N = the proportion of real negatives that test negative.
Read the whole thing to learn about other useful statistics, like "positive predictive value" and "negative predictive value", and the key role of overall prevalence in interpreting the results of such tests.
The key point is not that the tests are useless — they should be a crucial influence on decisions about who should be how stringently isolated. But it would be a disastrously bad idea to make those decisions under the assumption that the test results are 100% accurate.
A couple of previous posts on this general topic:
"(Mis-) Interpreting medical tests", 3/10/2014
"When 90% is 32%", 3/18/2014
Update — Christopher Weaver, "Questions About Accuracy of Coronavirus Tests Sow Worry: Experts believe nearly one in three infected patients are nevertheless getting negative test results", WSJ 4/2/2020:
The sensitivity of the currently available coronavirus tests seems to be lower than other similar tests, said Mike Lozano, a Tampa, Fla.,-based executive at Envision Healthcare Corp., a medical group that contracts to provide care at hospitals. Dr. Lozano said he estimates that the sensitivity of the tests is in the neighborhood of 70%, meaning nearly one in three positive patients walks away with a reassuring negative result.
bks said,
April 2, 2020 @ 10:09 am
(Tangent) At "This Week in Virology" virologists consider. but cannot answer, the question of whether some spoken languages produce more aerosols than others, and thus increase infectivity:
https://www.microbe.tv/twiv/
podcast #596, 1 April 2020
00:50:00
Thomas Hutcheson said,
April 2, 2020 @ 11:02 am
There is a difference in using testing to see whether to quarantine asymptomatic people and testing for diagnosis with symptoms. For purposes of R0 reduction, a test with a few false negatives does not matter.
cameron said,
April 2, 2020 @ 11:02 am
The accuracy of the tests is a minor detail in the context of how the pandemic is being handled in the US. Here in NY, you don't get tested at all unless you're so sick you need to be hospitalized. Thus, the vast majority of people who have symptoms and are likely infected are never tested, and hence never "confirmed" one way or another. Measures of sensitivity and specificity are minor quibbles given that most of those infected are never tested.
David L said,
April 2, 2020 @ 11:24 am
The study cited for evidence of the 70% false-positive rate is from China, published 2/26. I couldn't find details of the particular test used. I would guess accuracy depends very strongly on how many separate DNA characteristics of the virus are tested for. The CDC test that ran into trouble used three, but the test for one was found to be unreliable. The test developed in Germany and recommended by WHO is based on three DNA signatures.
Another issue: this author recommends CT chest scans as a diagnostic of COVID-19, but as we now know many people with the virus infection don't develop any symptoms, or only mild ones, so presumably wouldn't be diagnosable by CT scan. So this wouldn't be of any help in tracking the early stages of coronavirus spread through a community.
The point about testing accuracy is important, of course, but there are many other things going on here.
Leo said,
April 2, 2020 @ 11:59 am
This is not an even tangentially linguistic comment, but I would like to know how the sensitivity, specificity and so on of these tests are measured, and under what practical circumstances. In principle, to find out the accuracy of a testing kit you must know the true disease status of all subjects you test it on. But how do you determine that?
mg said,
April 2, 2020 @ 2:11 pm
From what I understand, a major reason for the very large false negative rate is because of the difficulty of swabbing adequately. Especially in people who don't yet have serious symptoms, it's easy for a swab to not touch any infected secretions. My guess is that the false positive rate is much smaller because it's less likely to misidentify what was actually caught on a swab.
So right now, it's safest to assume that yes means yes but no means maybe.
번하드 said,
April 2, 2020 @ 3:04 pm
@bks:
https://www.nature.com/articles/s41598-019-38808-z
"Aerosol emission and superemission during human speech increase with voice loudness"
[…] regardless of the language spoken (English, Spanish, Mandarin, or Arabic).
Just a data point so far.
AntC said,
April 2, 2020 @ 5:32 pm
the available tests for this condition seem to be even less accurate than medical tests generally are
@David L The study cited for evidence of the 70% false-positive rate is from China, published 2/26.
Beware: China has been 'generously'/'public-spiritedly' supplying tests to the world. Not so generous because they've been charging at commercial rates for them. More to the point, at least half a dozen countries have found them defective and returned them to the manufacturer.
Defective how? They're only 30% reliable/give many false negatives. That is, the patient does develop full COVID-19/if you re-test them with a German-manufactured test, it reads positive.
If PRC is using these same tests for its continuing monitoring of the disease, this would explain how numbers there have mysteriously levelled off. (That and that they have several times redefined their testing criteria.) This would align with the CCP's desired narrative that they have things under control. I believe nothing the CCP has claimed about the spread of the infection.
Why supply defective tests to the West? So that their government/public underplay the risks until their health systems become overwhelmed, and their economies tank, and PRC doesn't look so bad in comparison. Am I being a conspiracy theorist/couldn't this just be explained by incompetence and stupidity? wrt CCP those are inseparable.
Josh R said,
April 2, 2020 @ 8:06 pm
Leo said, "This is not an even tangentially linguistic comment, but I would like to know how the sensitivity, specificity and so on of these tests are measured, and under what practical circumstances. In principle, to find out the accuracy of a testing kit you must know the true disease status of all subjects you test it on. But how do you determine that?"
Controlled testing using sample populations for which their disease status has been confirmed by other means. How the confirmation is done can vary depending on the disease. In the case of Covid-19, as mentioned a CT scan can be done, as well as other more intensive blood and/or endocrinological analyses that are reliable, but not feasible for testing at the point of care.
번하드 said,
April 2, 2020 @ 9:13 pm
I also wonder whether the different rt-PCR test out there all test for the same RNA subsequences or (as I would guess) not. In the latter case one could try to use two different kits to improve confidence.
In quite different news, here's a tweak that might help a lot in a situation when you want to mass test a population and don't expect too many positives. (e.g. in Korea as of yesterday they had 414303 neg, 10062 pos, 18908 in process)
https://eurekalert.org/pub_releases/2020-03/guf-pto033020.php
Pool testing of SARS-CoV-02 samples increases worldwide test capacities many times over
Andrew Usher said,
April 2, 2020 @ 9:59 pm
It's already spread too far for anything to help much, and is going to infect most of the population no matter what we do. As shown above this drive for 'testing' is just asking to get ripped off by manufacturers of so-called tests. Remember that the coronavirus is just one more damn thing you can die of, not a unique existential threat (as our response to it might become).
k_over_hbarc at yahoo.com
AntC said,
April 3, 2020 @ 1:32 am
@Andrew U is going to infect most of the population no matter what we do.
Yes, but it matters when/how quickly it spreads. And no, not so many people have to die of it as Italy is experiencing.
1. If a country can delay the spread enough, perhaps better treatments can be identified. That's not a 'cure' or a vaccine, but a way to ameliorate the worst effects.
2. Italy/Spain's health systems are overwhelmed to the extent they're rationing/triaging ventilators. People are dying 'unnecessarily' in the sense if ventilators were plentiful (and the staffing to manage patients) more patients would survive.
3. Timely testing and contact-chasing with isolation will reduce the pace of spread/avoid health systems getting so overwhelmed. (In the U.S. it's probably too late; by all means let the virus rip — you're welcome to catch it.)
4. There's anyway going to be the usual demand on health systems from heart attacks/cancer/diabetes/respiratory diseases/whatever. If the health system is overwhelmed/refocussed on COVID-19, a proportion of those cases will result in 'avoidable' deaths.
COVID-19 is on track to cause more deaths than usually attributed to flu/colds. Sure those patients probably have weakened immune systems anyway and would die 'prematurely' one day. Explain that to your elderly relative on their deathbed, that it's the cost of developing 'herd immunity'. You're coming across as callous.
Not everything about 'big Pharma' is evil. Indeed I'm not sure it's only 'big Pharma' supplying the tests; it's certainly not only them supplying ventilators.
Philip Taylor said,
April 3, 2020 @ 5:31 am
I am very pleased to be able to confirm Ant's last statement above : "it's certainly not only [big Pharma] supplying ventilators". A small local company with which I have a connection are indeed "provid[ing] vital support to the [UK] government's ventilator challenge".
Andrew Usher said,
April 3, 2020 @ 6:42 am
AntC:
As you mentioned it, I must remind you that I and probably everyone have seen elderly relatives die of a number of things, and you don't know my own history in the matter. It is unavoidable, but you are implying that we should regard deaths attributable to COVID as worse than any other deaths, which I certainly can't accept. It would indeed seem callous to mention it in one specific case in the manner you stated, but to talk about in in general is not more so than any discussion of mortality is. (Another things that does strike me as possibly callous is appearing to use the disease to make unrelated political points, which I carefully avoid.)
Like it or not, 'herd immunity' is a thing and by whatever name needs to be mentioned in any rational discussion. Now, I would rather not have had this happen, I am not sitting thinking 'Wow, this is great, we can thin the useless old and sick people'. But it did happen, and if anyone could have prevented it it was China when it was first discovered but instead they denied and hid it until they could no longer, ensuring it would spread to the rest of the world. Now we deal with that fact.
I never mentioned the phrase 'big pharma' not do I here have the beliefs associated with that term; I was responding to your earlier comment about China supplying defective tests. US manufacturers are less likely to give trouble because they might actually face consequences, but any profit-driven business may be tempted.
The exact death rate or number is not my concern; no one can predict it currently. What's important is the general fact that it is low based on number of people exposed, and especially so for healthy people. So the virus itself will never wreck our society, while our response could. It's not 'just the economy' – first, nothing is ever 'just the economy' because 'the economy' represents real wealth, which is real goods and services that hopefully at least improve people's lives even if not essential – nor is it just the disruption, though that will be tremendous. It is the plain fact that as our lives are not of unlimited duration, we all lose a fraction of our useful life to the disruption, as if it were a partial death. For myself, I reckon that the chance of death if I get it (if I have not already) is less than the fractional loss of life I will probably incur due to the shutdown, so I should prefer the former – and averaging this across the population shows that it's quite possible shutdowns are a bad idea even if they work at completely wiping out the virus (which is not likely).
I do agree that health systems being 'overwhelmed' is a legitimate concern, though almost surely exaggerated as they can or should be able to improvise to mitigate any such thing (as is already occurring here). However the policies being adopted do not address this only but are overly broad and effectively have indefinite duration with no plans for how they will end, and that open-ended nature makes it also impossible for businesses and other organisations to plan. This can't be accepted as a defence.
I have not even yet mentioned that freedom itself is a great value, and that the consequence of that is obvious – it can only make my case stronger.
Peter Grubtal said,
April 3, 2020 @ 10:44 am
AntC. :
The extremely drastic, unprecedented measures taken to "flatten the curve" will have their own effect on mortality. The economic downturn alone is bound to have a negative effect, with fewer resources being available for, inter alia, health care.
The effects will likely be delayed action, indirect, diffuse and difficult to quantify. But I wouldn't be surprised if in human life-years lost it will be greater than those due to corona.
AntC said,
April 3, 2020 @ 5:32 pm
if anyone could have prevented it it was China when it was first discovered
I'm not following your logic. COVID-19 is more 'successful' in transmission than SARS 2003 or H1N1 2009. China could not have "prevented" the eventual spread (except perhaps by immediately closing its borders for ever — even in December, because the virus is 'sneaky', probably cases had already escaped to the world). So by your logic, the worldwide deaths would have happened anyway, so what benefit would earlier action by PRC have rendered in the long run? The totalitarian lockdown in Wuhan/Hubei was as pointless as all the other lockdowns around the world, by your thinking.
I'm in New Zealand, currently in 4-week total lockdown, currently 1 death attributed to COVID-19 (acquired overseas) in a population of 4.5 million. More impressively, I was recently in Taiwan, limited lockdown (for example all the schools are still running; all restaurants are open with distancing arrangements amongst the diners), 5 deaths and much fewer cases than NZ, population 23 million, very dense housing, high-volume interaction with the Chinese mainland up to the outbreak. (To link back to the OP here, both countries (NZ rather belatedly) are testing everybody remotely connected to known cases, whether or not they show symptoms; and retesting them; and retesting them.)
Yes there are/will be severe economic impacts on both countries. Never the less their populations are largely supportive because they can see how much worse it is in other countries.
I can't make a judgment with reference to (say) the U.S. or U.K. Both governments 'missed the window'. They'll both suffer large numbers of deaths and compromised health outcomes from COVID-19; and large disruption to their health systems generally, causing further deaths. (That is, if whatever's in the U.S. counts as a 'health system' — it's rather a joke by world standards.) Is it 'worth' at this late stage piling personal and economic disruption on top of the deaths? Perhaps your 'full steam ahead, never mind the torpedoes' is an appropriate response: save the economy, let the physically weak perish. Ayn Rand would be proud of you! I emigrated to NZ from Britain precisely because here is a kinder polity.
AntC said,
April 3, 2020 @ 5:48 pm
you [AntC] are implying that we should regard deaths attributable to COVID as worse than any other deaths,
No. (Or actually, dying alone in a ventilator or on the floor in a hospital corridor with no chance to be with your loved ones, or they with you because they're locked down in another city — I mean Yes.) I'm saying a proportion of those deaths could be avoided by careful management and/or health technology (including treatments) and adequate staffing, without health workers themselves getting invalided-out with the virus.
"avoided" is of course a misleading way to say 'delayed' — we're all going to die eventually. So do you advocate strangling at birth kids with congenital defects (who are never going to be able to contribute to the economy); or those born into poor circumstances who statistically will be unable to afford their own healthcare over their lifetime?
Christian Weisgerber said,
April 3, 2020 @ 6:24 pm
@bks
Ban aspirated stops!
Andrew Usher said,
April 3, 2020 @ 10:00 pm
Peter Grubtal:
Yes, you can look at it that way as well – making it even worse.
AntC:
I did not say China definitely could have prevented it, only that if anyone could, it was them – i.e. no one else could, because it had spread too far. They chose however to delay; I'm not here going to speculate why.
If you are correct about how well the policy is working for your country it may be justifiable there – but you yourself admit that the US has gotten too far in for that.
You seem to have an incorrect idea of my political beliefs. If you'd read all my posts here over the years, you couldn't – I am an economic socialist, and have been since I figured out many of the things I'd been told about the capitalist system were BS. I absolutely believe our health system is a relative embarrassment, and I have always supported some form of nationalised system and have said so at every opportunity. No one should be born into a position where they won't have access to health care or any other essential to decent human life.
But that is not particularly relevant now that we are addressing this current epidemic, because you go to war with the army you have – we must address this with the system we have. Of course the hospital system does have a role in reducing deaths from this as from anything else, and I know ours are trying, but I'm not an expert in that.
The correct choice for us should be clear from what has been said; I am far from the only American that's said this, and even more think it but won't say it in public.
Christian Weisgerber:
I guess I'm doing my part then, I aspirate my stops less than most … (seriously I think any sort of speaking is about the same hazard)
Adam said,
April 4, 2020 @ 12:41 am
It is very important to distinguish false positive and false negative here. RT-PCR assays almost never have false-positives (unless the operator makes a mistake). False positives would make RT-PCR useless for its current task of finding new cases. Australia has tested more than 280,000 people. If we had a 10% false positive rate it would look like we have 28,000 cases. But actually we only have 5,500 cases.
False negatives means you might miss it the first time. But your chances of missing it after two tests is 10%, and 1% for three tests. So you will know eventually. I much prefer to have a 30% chance of false negatives and a negible chance of a false positive than the reverse situation.
AntC said,
April 4, 2020 @ 1:29 am
You seem to have an incorrect idea of my political beliefs. If you'd read all my posts here over the years, you couldn't – I am an economic socialist, …
Andrew, on a point of information, "socialist" in a U.S. context means something very different to in Europe or most other liberal democracies. By those standards, Bernie Sanders is to the right of most Right-Centrist parties — which all accept without a murmur there should be publicly-funded universal healthcare, free at the point of delivery.
So your believing yourself to be U.S.-"socialist" is entirely consistent with me taking you to be Libertarian/callous 'let the weak perish'. (And I have formed that judgment of you from not just this thread.) Furthermore your views chime very much with what Trump has been saying up to recently, with what Dominic Cummings is alleged to have opined in the U.K., and what several Republican State Governors seem to still believe.
Of course we can each claim our views align with a silent majority — be it Americans or NZ'ers. I note that both in NZ and in Taiwan, the official oppositions (parties to the right of the respective current governments but to the left of Bernie Sanders) are fully supportive of the current draconian arrangements.
The only politicians in the U.S. who are publicly voicing opinions like yours are the likes of Rand Paul (now invalided out, ironically). You might wish to reconsider where you are on the political spectrum.
Andrew Usher said,
April 4, 2020 @ 9:27 am
I tried to avoid it, but you've changed the subject to US politics. I guess I'll start by following. No, I don't accept that 'socialist' means different things in different countries; 'liberal' and 'conservative' are relative terms, yes, but not 'socialist', not in any manner I'd consider correct. I don't normally like to give a precise definition as I'm not really interested in quibbling about words, but if I were to do it in one sentence, I'd say "the belief in economic justice and that government has a role in upholding it" – the phrase 'economic socialist' that I used is actually redundant to me but I used it to try to avoid other meanings people attach to the word.
I called myself socialist before I heard of Bernie Sanders; it's true, indeed, that I would support him over all the other candidates at this election and the last, but my beliefs are not determined by any one person or party. I'm not particularly interested on being _on_ the political spectrum at all, so I don't consider it.
You call me callous – well it's pretty easy to be smug on your high horse from your little island country that you believe (rightly or not) won't be seriously hit. I know how Americans really suffer from the current shutdowns, and I know that it's going to get worse if there's no plan for ending them (as there isn't). It's not a matter of weighing lives against 'the economy', but against other lives – and what is 'the economy', really, if not just an abstraction for the goods and services real people use? Yes, I admit that our present economy helps the rich and large corporations more than it should, but that's not the issue right now. Right now, when talking about response to this crisis, is not the time for partisan politics or 'shoulda, woulda, coulda'. I wish that this weren't happening, of course, I wish no one had to perish at all, but that's just not the fact.
I am the one standing for the interest of ordinary people, as a socialist should; you are standing at best for a moral principle that isn't and can't be applied consistenly – as I'd expect from a religious leader. True courage means knowing that sometimes you have to stop trying and cut your losses; true leadership means knowing that sometimes trade-offs must be made. One can't in any decency take away for a long time the freedom of hundreds of millions in an attempt to save hundreds of thousands, with no higher cause at stake.
[As for Rand Paul, he is not sick but only following standard isolation procedure. Any personal remarks about him would not be warranted.]
AntC said,
April 4, 2020 @ 6:44 pm
Andrew Rand Paul spread the virus throughout the Senate building and Kentucky, and many points in between, before quarantining himself. Other senators representing the same or lesser risk have behaved like decent human beings. "Not sick" does not mean not infectious/not liable to kill others he comes into contact with.
You'll be no doubt delighted to hear 'only' 581 deaths in Italy yesterday; they're turning a corner.
I (as an ordinary person/humanist, no sort of leader) find your views repellent and repulsive, irrespective of what you call yourself. Since this is by now in no way on-topic for LLog, I suggest myl deletes all of my comments, and yours.
Andrew Usher said,
April 5, 2020 @ 12:02 am
I certainly do not believe any posts should be deleted; you have effectively conceded the argument and should not be able to hide the evidence. I will not respond to any further points when it is clear you have no interest in discussion. Your anti-American attacks and inability to refrain from partisan politics should speak for themselves.
(This entire thread starting with the post was technically off-topic here.)
Andrew Usher said,
April 5, 2020 @ 7:57 am
Well, I can count that one post was deleted. While it doesn't seem to be one of mine it still disturbs me that the record can be silently altered like that (obviously there are good reasons but I didn't see any here).
Adam:
I don't think false negatives are going to be completely uncorrelated like that. Whatever factor caused a negative outcome the first time (unless it's a flaw in the test) is likely to persist on further testing. None the less I can see here why false negatives would be preferred to false positives; for other kinds of medical tests it's definitely the contrary!
James Wimberley said,
April 5, 2020 @ 10:25 am
My son is an English teacher in Korea. He reminded me that one key to Korea's success is tracking, and added one detail new to me. When a new outbreak cluster is identified – say a particular shop – , its location is immediately made public. Anybody who may have been exposed there can then get themselves tested for free. Less Confucian societies are going to have to adopt such losses of privacy when the epidemic recedes and general quarantines end.
Philip Taylor said,
April 5, 2020 @ 3:21 pm
There are parts of the UK that are crying out for such supposèd "losses of privacy", James. Here in Cornwall, for example, we are told only that there are $n$ cases of Covid-19 know to exist, and no more accurate information (in terms of locality) is publicly available. Cornwall is almost 1400 square miles in area, with a coastline of over 400 miles, and well over 100 miles in length. Compare that with (e.g.,) Greater London, with an area of just over 600 square miles (less than half the area of Cornwall), yet for which 30 different localities are able to report independent statistics. If we knew (for example) how many cases per head of population there were in St Austell, and had the same statistic for Bodmin, then we would know where was probably safer to shop. As it is, we just have to shop and hope (or, for those of us of a certain age, ask others to shop for us and leave the goods outside).
Andrew Usher said,
April 6, 2020 @ 6:54 am
I'm not sure if such things should really be called losses of privacy, but if everyone found it where it would be 'safer', then that place would no longer be safer with all the people there! You really can't run away from it, not like that.
I don't think I'd have any problem with identifying locations or events where coronavirus was spread, but such announcements would need to be rather prompt to be effective.