Continuing the discussion from Is there really a huge spike of covid patients in the NHS? Apparently not:
Hi all,
this is an attempt to try and gather together a few thoughts around the subject of the PCR tests, false positives and positive tests vs “cases”. This subject has come up a lot on this board (and in many online spaces) and - personally - I’ve had an interesting and challenging journey through this topic, so I thought I’d share some of my thoughts for folks here to mull over and critique.
Some important references that are worth reading and which I’ll be basing some of my thinking on are listed at the bottom. Thanks to everyone here who brought all these pieces to my attention - I do appreciate it.
I am always, and I mean always, ready to believe that I might have misunderstood something, or just got something plain wrong. If I have made mistakes in my thinking here, I would be happy to be corrected.
Bottom line on top
This might be definitely will be a long post, so to save those who are not interested in the details, I’ll just state up-front what my conclusions are.
1 - I used to think that false positive PCR results would be a problem, but I no longer think so. A false positive is someone who gets a positive result from a PCR test, but in reality has not ever been exposed to the virus. I think the people who worry about the number of false positives might be overestimating the true number by a factor of up to 40,000.
2 - I think the false positive rate for PCR and SC2 is close to 0.005% or most likely smaller. This is based on a joint research project ongoing for at least 6 months by the ONS, University of Manchester, University of Oxford, Public Health England and the Wellcome Trust [1]. That is much better than the critics portray it to be. And it is quite believable, then, that in certain circumstances, a mass test could throw up zero or only a small handful of positive results.
3 - I do believe that a positive PCR test doesn’t necessarily indicate any of the following
- The person is sick, or will become sick
- The person is contagious or will become contagious
- The person has been sick in the past
4 - Despite point 3 above, I still do believe that the PCR test is one of the best tools we have to try and manage this illness (Covid-19) and the spread of the virus (Sars-Cov-2)
I’m going to start with accuracy, and then move on to the subject of what the meaning or relevance of the test is.
Accuracy and false positives
Actually we have several folk who have given a pretty good demonstration of how the accuracy of a medical test affects the number of positive results. This talk [2] that @PatB found, is a pretty good layman’s version. Pieces by Heneghan [3] and Yeadon [4] (thanks @Bob_sYourUncle ) also do a pretty decent job of working through the logic. Unfortunately each of these pieces, whilst being strong on the concepts, are terrible on the detail, and it’s worth seeing why that is.
There are three important bits of information regarding the accuracy of a test:
- The Sensitivity of the test (false negatives)
- The Specificity of the test (false positives)
- The prevalence of the illness in the population being tested
For the purposes of this discussion, we can ignore the sensitivity - apparently no one on this board other than myself is concerned about false negatives. I think that’s a problem - but that’s just me.
On the other two parameters - the specificity and the prevalence, all the people I’ve mentioned above get it badly wrong.
As has been discussed, Bayes’ formula is the way forward here. The crucial thing about Bayes’ formula for non-mathematicians is that it gives a way to use knowledge about how widespread a disease is, in the determination of how likely a positive test is to be truthful. For diseases that are incredibly rare, it’s unlikely any particular individual has that disease even if they test positive for it. Imagine a situation where a disease was so rare that only 1 person on the planet had it (let’s call that person “Donald Trump”). If 10 new people went and got tested, and came back positive, the probability is strong that there is still only 1 Donald Trump, and the tests were false positives. No matter how good the test was.
The important point about Bayes Theorem is that we have to multiply together the specificity by the prevalence of the illness. For those who are interested in the maths, you can see a couple examples below to get a feeling for this. For those who couldn’t give a monkeys, please skip over this part to the part below.
mathematical aside
Bayes Theorem says that, P(A | B) ~ P(B | A) x P(A)
A = clear (no virus), B = +ve test, and P means “probability of”. So re-wording the formula, we get:Prob(clear despite +ve test) ~ Prob(+ve test despite being clear) x Prob(being clear)
That is the probability of false positives is proportional to
P(false positive) ~ (1-specificity) x (1-prevalence).A quick sanity check tells us we’re on the right track here:
100% specificity
In this case, we get the probability for a false positive ~ 0 x P(1-prevalence) = 0
If the test is perfect, we get zero false positives100% prevalence
If everyone is sick with the virus - 100% of the population - then the probability of a false positive is ~ (1-specificity) x 0 = 0.
If everyone has it, then the chance that you have had a false positive is zero.
From the formula and examples above we can say the following
- as the test specificity goes down, the probability of a false positive goes up
- as the prevalence in the population drops the probability of a false positive goes up
Pretty common sense so far. Where the folk mentioned above go wrong is by
- greatly underestimating the specificity of the PCR test, and also
- greatly underestimating the prevalence
Two big mistakes that get multiplied together to make a really big mistake.
PCR Specificity
All of the people mentioned in the references tend to assume the specificity of the PCR test for Sars-Cov-2 to be 95 - 99%. That is 1 in 20 or 1 in 100 on average get a false positive. That is literally hundreds of times too low. Maybe worse than that. As I mentioned above, the best study I’ve found so far is a joint effort by the ONS, Oxford Uni, Manchester Uni, PHE and the Wellcome Trust [1]. They say (in the section on Sensitivity and Specificity):
While we do not know the true sensitivity and specificity of the test, our data and related studies provide an indication of what these are likely to be. In particular, the data suggest that the false-positive rate is very low, under 0.005%.
That gives a specificity higher than 99.995%. Not 95%. This difference alone is enormous on the false positive rate, but it gets even worse.
Prevalence
This is a subtle point, but important. The people quoted above generally regard the prevalence of SC2 to be very low in the community - 0.05 - 0.1%. They might be right about that, and obviously it’s changing all the time, but the point is that this is misleading, and not the right number to use in this case.
Tests are not handed out at random to people in the population. This was particularly true at the beginning when very few people could get tests at all. Even today, the percentage of random folk who just up and decide to get a test for no reason, and are successful at that, is very low. People get tested when they or their doctors have some strong reason to believe that they have come into contact with the virus. This is not a random sample of the population.
A more useful number from the point of view of false positives and Bayes’ theorem, would be to ask - what is the prevalence of the illness in the sub-population that is actually going to get tested?. Happily we have access to some of that data, and it varies with the number of tests being done. The important number is the ratio of positive tests to total tests being done. That is a more correct number to use in Bayes theorem, and that number has varied between 2-40%, depending how many tests were being done. Let’s just use the current number of 2% and ignore the 40%.
Conclusions on accuracy and false positives
Comparing the two important parameters, we see that the critics who worry about false positives underestimate the specificity by 200-1000 times, and underestimate the prevalence by 20-40 times.
We have to multiply those numbers together to get the full scale of the issue. I reckon they are overestimating false positives by a factor of 4000 - 40,000 or even worse!
That’s a big mistake to make. Of course, these are rough numbers and a simple back-of-the-envelope calculation. The problem might not be overstated to such an extent, but it is definitely nowhere near the size of problem that these folks are telling us that it is. Certainly several orders of magnitude less of a problem.
So what makes me think I’m right? Well several things:
- The ONS study mentioned has a decent discussion on why they think their specificity number is correct. There is a companion paper [5] that goes into more detail.
- It’s also the case that studies have shown that most people (99.6% in one example) who test positive with PCR go on to develop antibodies to the virus. That would not be the case if 88% of PCR tests were false positives [6]
- Other non PCR measures of the spread of the virus (the ZOE symptom tracker) agree with the numbers that come out of the PCR test.
- Several countries reporting low or zero cases despite decent testing regimes (and yes, @Dimac - I’m including Australia here) show that the specificity of the PCR test must be high
- The death rates would be astronomical for this virus if we have had over 100,000 deaths and only 18% of the cases were true. That would make the death rate maybe 5 times higher which we would see around the world in other studies.
As far as I can see, the false positive problem for PCR and targeted testing is simply not a problem anything like the scale it’s blown up to be. It would be different for different tests, and a moonshot style full-spectrum test strategy, but we’re clearly never going to do that.
Interpreting PCR positive tests
Ok. Let’s take a look at the other part of the issue around PCR testing - what do they actually mean, and what should we infer from them?
As far as I understand it, the PCR test simply looks for a piece of viral RNA in a sample from a person. If the test is able to match that piece of RNA with something in the sample, then the test comes back positive, otherwise negative. The sensitivity of the test increases exponentially the more cycles that are run, so there is the potential to magnify up tiny fragments of virus in the sample.
A positive test, then, simply means a match has been found to a benchmark bit of RNA. Depending how specific the benchmark is to the virus you’re looking for, you can be more or less certain that the person with a positive test has at some point come into contact with the virus. If the benchmark is perfectly unique, then it can only have come from that particular virus. If the benchmark is common to a variety of viruses, then you could be matching to any one of them.
That’s what I understand by a positive result in the PCR test. If I’ve got that wrong, then I am more than happy to be enlightened further.
So, the obvious question is what is the connection between a positive PCR test and (a) the probability of going on to get sick with Covid-19? or (b) actively being infectious and able to spread the virus to others?
As far as I know, the answer to both of those questions is the same - we just don’t really know. We don’t know if you will go on to get sick after testing positive. We don’t know if you are or will become infectious after having a positive test result.
As @PatB has pointed out to me several time, quite correctly, even the creator of the test has gone on record to say that it is useless for diagnosing illness. Fair enough - I am certainly not in any way qualified to argue with that. Let’s just go ahead and accept all of these caveats.
The question is what should we, as a community, do when people test positive with PCR for SC2? Let’s consider the problem like this:
You’ve just got a positive PCR test result. This means that you might go on to get sick. You might be contagious. Or perhaps neither of those things. How do we know what actually will happen? Literally, the only way to know what is going to happen is to "wait and see"™ (@RhisiartGwilym your little phrase does pop up everywhere).
So, waiting perhaps 10 days will demonstrate for sure whether you are going to get sick, whether you are infectious or whether it’s a false alarm health-wise and pandemic-wise.
Therefore, despite all the acknowledged uncertainties around PCR, acknowledging the lack of direct connection between PCR and illness, the lack of direct connection between PCR and contagiousness, it still seems to me that the best thing to do for the community in which you live is to self-isolate for 10 days upon receiving a positive PCR test.
I think that alone (with all the other support that the Indy SAGE team suggest) would be enough to wallop down the virus to the point where everyone else can live more or less normal lives. Does it mean you have Covid if you test positive? No. Does it mean you’re contagious? No. Does it mean you might be? YES! So, out of caution and a desire to stop transmission, you pay a 10 day price and then you’re out.
Despite all the critics of PCR this seems like the most reasonable thing on Earth to me. Better, by far, than any of the alternatives I’ve seen.
Therefore, yes - I DO support PCR testing, I do take it to be the best we can do. I don’t think false positives are a problem and I do think people are worrying way too much about it.
Does it mean that it can’t be used as a political football - no. Of course it can. But the Great Barrington declaration is also a political football. Lockdowns are too. Almost everything is. In the meantime we need some way of managing the virus, and I can’t see a better way than PCR.
So that’s it. Congratulations to any intrepid reader who made it down this far. I warned you at the beginning it would be long. If I’ve made any mistakes or got things plain wrong just let me know. Any thoughts or comments or critiques are welcome - and thanks to all who gave me the information to navigate this subject.
Cheers
PP
References
[1] Coronavirus (COVID-19) Infection Survey, UK - Office for National Statistics
[2] Joseph Nucara PCR Testing Factual or Fraudulent - altCensored
[3] How many Covid diagnoses are false positives? | The Spectator
[4] Lies, Damned Lies and Health Statistics – the Deadly Danger of False Positives – Lockdown Sceptics
[5] https://www.medrxiv.org/content/10.1101/2020.10.25.20219048v1
[6] https://www.thelancet.com/journals/lanmic/article/PIIS2666-5247(20)30120-8/fulltext