Apples To Apples

How do you figure out how bad COVID-19 is compared to the flu?  It's harder than it looks and not just because we are right in the middle of a pandemic.

Like many, I've spent much time looking at COVID-19 statistics and medical literature trying to get a handle on what is going on.  One common topic has been determining the Cumulative Fatality Rate (CFR) for the illness, a rate derived from the ratio of deaths attributed to COVID-19 over the number of individual infected by the virus.  The flu season CFR, as calculated by the CDC, is used as a reference point to determine relative severity of the COVID pandemic.

Let's start with some basic information from the CDC website regarding the flu seasons from 2010-11 through 2018-19.  The CDC provides information on numbers of flu cases, hospitalizations, and deaths for each season.  2013-14 with an estimated 29.7 million cases and 38,000 deaths yields a calculated CFR of 0.128.  The worst year in terms of CFR is 2014-15 with a rate of 0.169, and the highest fatality total was in 2017-18 (when I had the flu for the first time in decades) with 61,000 deaths and CFR of 0.136, very close to the overall average for all nine years.

Now let's look at the current COVID-19 data for the United States, courtesy of worldometer.  We have 819,146 total cases and 45,340 deaths for a current CFR of about 5.50 or nearly 40 times the average flu season.  So, we're done, right?  Uh, no.

First, we have the temporal issue.  We are in the middle of the pandemic and don't know what final numbers may look like.  Another temporal issue is also significant.  Flu season runs for six months and the average number of deaths over the past nine seasons is 38,000.  We've had 45,000 COVID-19 deaths in a month.

But there's another, much bigger, problem.  Everyone acknowledges that the total case amount for COVID is an underestimate.  In many parts of the U.S., people with COVID symptoms, but not in obvious distress, are being turned away without testing because of a lack of capacity.  Further, we've become aware that a large percentage of COVID infections, particularly among younger people, are asymptomatic; either no symptoms or mild symptoms they mistake for allergies or a cold.  So how do we figure out a denominator?  We don't know at this point and it's led to a lot of discussion and argument.  Everyone seems to agree that 5.5 is not a realistic CFR and many experts think it is 1.0 or less.  A recent study done by Stanford scientists in Santa Clara County claims that the infection rate (symptomatic and asymptomatic) is 50 to 85 times higher than shown by current testing.  If you extrapolate the 85X estimate to the United States you have 68 million cases and a CFR of 0.067, well below the CFR for the mildest flu season.  Today it was announced that in New York City 21% of the population may have been infected.  So, now we're done, right? Uh, no.

The Stanford study, released in the past week, is controversial and a lot of people who know more than I about the study say there are significant defects in it, and we have yet to hear what types of critiques will be made of the New York data.

And in looking at this issue a couple of days ago, I also came to the realization we have not been talking about an apples to apples comparison.  The focus with COVID-19 has been on trying to measure all cases, symptomatic and asymptomatic as in the Santa Clara and New York studies, and, for some, to compare them to flu cases.  I've seen some arguing that since they believe there are a huge number of these undetected cases out there the CFR for COVID-19 is no worse than a bad flu season.

In researching this question, I was looking at the CDC website and realized the CDC does not include asymptomatic flu cases in its case estimates (read the How Many People Get Sick With The Flu Every Year? section).  Therefore attempts to compare all COVID cases to flu cases is not an apples to apples comparison.  Further, I was surprised to learn that a large percentage of flu cases are asymptomatic, though there is a wide range of estimates in the medical literature.  As best I could determine the CDC estimates that perhaps 50% of flu cases are asymptomatic, though there are also peer reviewed studies putting the percentage as high as 77%.  Here's the specific language from the CDC study:

Because of its greater clinical relevance, we studied symptomatic influenza infection. Estimates of the percentage of influenza infections that are asymptomatic include a common approximation of 50% [33], 33% in 1 review [34] and 4%–28%, 0–100%, and 65%–85% in another review [35]. Both asymptomatic and symptomatic infections are captured in serological studies. The commonly cited “5%–20%” figure came from a serological study [5], and so represents both symptomatic and asymptomatic disease among a mix of vaccinated and unvaccinated persons. If 50% of influenza were symptomatic, this would correspond to “2.5%–10%” with symptomatic disease, which is very similar to the range that we report.

If you re-examine the CDC data on flu seasons from 2010-11 to 2018-19 in light of this information, and including asymptomatic and symptomatic cases using the 50% assumption, the CFR for the worst flu season drops from 0.169 to 0.085, for the average season from 0.135 to 0.068, and for the mildest from 0.096 to 0.048.

[For purposes of analysis I am setting aside that the CDC methodology for determining the number of flu cases and deaths is primarily based upon modeling assumptions, not test data, whereas a substantial portion of the COVID-19 death data is based upon actual test results and direct clinical observation of symptoms.  So while there is a range of uncertainty regarding the actual COVID-19 death count, the range is much more uncertain for the flu.]

And where does that leave us with COVID-19?  It remains unknowable what the final death count will be; we don't even know if there will be further waves after this one subsides, as there was with the Asian Flu, which begin in the summer of 1957 and finished with a last wave in the spring of 1958. But what looks likely is COVID-19 is going to be substantially more deadly.

Why?

We are still in the peak period (I hope it gets no worse) for deaths, averaging 2,000 a day since April 7.  In the fewer than 48 hours since I began writing this post and used the U.S. data in the 4th paragraph, total cases have increased by by 57,028 to 876,174 and deaths by 4,311 to 49,651.  We've had 49,471 deaths in the past month.  Flu season lasts six months and the highest total is 61,000.

Based on what we are seeing in European countries once the peak has passed there remains a long, slow decline in deaths.  It is likely the United States will see 75,000 to 80,000 deaths in this wave.

If you go back to the top range 85X estimate from the Santa Clara study and estimate 75,000 deaths you would have a 0.087 CFR for COVID, about that of the most deadly flu season in the past decade.  However, the 85X estimate and the CFR are unlikely to stand up.  Even since I began writing this post, additional criticisms of its methodology and statistical analysis have been mounting.  Moreover, the 85X estimate is contradicted by the real life experiment running in New York City.

From March 11 through April 22 the NYC Department of Health reports 10,290 confirmed COVID-19 deaths and 5,121 probable deaths.  The population of New York City is about 8.4 million.  Even if there were no additional deaths in the city, the CFR, using the entire population infected and non-infected, and based on confirmed cases is 0.133 or 1.5x the worst flu season or, if probable cases are included it rises to 0.188 or more than 2x the worst flu season.

Moreover, NYC now has 144,555 confirmed cases.  If the infected rate was actually 85x higher that would mean more than 12 million cases or 3+ million more than the population of the city.  Even at 50% there would be nearly 7.3 million cases meaning a nearly 90% infection rate, which no one thinks likely.

If the 21% estimate unveiled today proves to be accurate, it translates to about 1.79 million COVID infections in the city and a current CFR, depending upon whether you include probable cases, of 0.58 to 0.86 or 7x to 10x the worst flu season.

The NYC data also helps refute another myth that has arisen, that somehow overall total deaths are not higher even with COVID.  The linked page above shows 15,411 confirmed or probable COVID deaths along with 10,326 other deaths during the same time period or 25,737 in total.

The latest annual mortality data in NYC shows about 57,000 deaths a year.  For purposes of this analysis let's round that up to 60,000.  The time period for the city's data covers 43 days or 11.8% of the year.  Let's round that up to 12%.  If deaths were normally evenly distributed during the year our expectation would be about 7,200 deaths.  But maybe they are not so evenly distributed, so let's add 20% to the total to make it 8,640.  Even with that adjustment the actual number of deaths in NYC are 3 times normal. 

And all this doesn't even get into evaluating how much the steps taken by people and government to avoid getting infected have reduced cases and deaths compared to the flu, or that for the flu we have an annual vaccine that, to some extent depending on the year, also reduces the number of flu cases.

We don't know how much more deadly than the flu COVID-19 will ultimately be, but we are not in usual times, and COVID-19 is not the flu.