Metrics

Running a corporate metrics program for environmental and safety performance for a dozen years taught me a few lessons and following covid metrics for the past two years has reminded me of those days.

There are a number of factors important in designing and operating a metrics system.  Here are a few off the top of my head.  I'm sure I'm forgetting a few (age, you know).

1.   What is it you are trying to measure? 

2.   Are there alternative measures that might be more effective?

3.   Why are you trying to measure it? 

4.   What is the value of collecting the data versus the collective effort needed to collect it?

5.   How difficult is it to measure accurately?

6.   How can you validate the metrics being reported to you for accuracy?

7.   What is the criteria for accuracy?  Is close, good enough?

8.   Who will be the customers of that data?  Different customers may use it for different purposes.

9.   Are there desirable behaviors you trying to drive with the metric?

10.  Could you drive undesirable behaviors or create unintended consequences with the metric?

11.  Are you periodically reviewing metrics to make sure they are still necessary and driving the right behavior?

There are two other basic concepts to keep in mind anytime you are measuring and comparing performance on a metric:

Are you using the same measuring stick in all your operations and are others doing the same?

For instance, in the case of covid, is everyone using the same definition of "covid death"?  If not, comparisons become difficult.

Note: Even if it is not the "right" metric if all are using the same measuring stick you can still make useful comparisons.

Even if you are nominally using the same measuring stick is everyone counting accurately?  

In the case of international covid comparisons, some countries may have deficiencies in their health care and data collection systems that make accurate counting difficult, while others may be deliberating manipulating the data.  The global measurement problems with covid are way beyond anything I encountered in the corporate world; The Economist estimates the actual death toll may be up to 3X the current officially reported 5.5 million.

Which brings me to the U.S. metrics around Covid, a situation that has always presented challenges but is becoming even more difficult with Omicron. 

Since the start of the pandemic the states have been reporting Covid hospitalizations.  Under the CDC definition these include all those admitted, for whatever reason, who are found to have Covid.  This is a useful metric in gauging how widespread the pandemic is.  However, it does not tell you how many patients have been admitted because of Covid which would also be a useful figure, particularly for the public.  In my opinion, both types of hospitalization data should have been reported.

This information has been difficult to find.  Over a year ago, I did some research and concluded that about 30% of Covid hospitalizations were for reasons other than Covid.  Subsequent studies indicated that, for children, about 40% of such hospitalizations were for reasons other than Covid, a finding that makes sense given children have less serious Covid health outcomes than adults.

Because Omicron is so easily transmissible, the number of cases has increased very quickly and I would expect the number of hospitalizations with covid to increase substantially but the data I've seen from various hospitals and areas in the U.S. and elsewhere is that up to 70% of those hospitalizations are not because of covid.

The scale and pace of the increase is enormous.  Globally, the 7 day average pre-Omicron was about 490,000 cases (with the pandemic high being 826,000) while it is now 2,647,000 (source: Worldometer).  For the U.S. in early December the 7 day average was 71,000 (with the high in early 2021 at 251,000) ; it now stands at 789,000.  It's probably a good starting assumption now that anyone admitted to the hospital for any reason is very likely to have covid, or to get it while hospitalized.

Another complicating factor is that much of the Northeast, Midwest, and Southwest were still in the Delta wave when Omicron hit and since hospitalizations and deaths lag cases by 2-4 weeks it is difficult to distinguish between the two in the metrics (although individual studies have affirmed that outcomes are significantly better for Omicron).  In the case of Arizona, the case and testing positivity data show that Omicron hit us the week of December 27 which means hospitalization and, more importantly, mortality data is probably still predominantly Delta related.

This also means that covid deaths may become harder to assess.  To be precise, because so many of those hospitalized and in ICUs will have covid, determining whether covid is actually the factor, or a contributing factor to those deaths will become more difficult and I think there will be more people in that category.  Since those dying with a positive covid test are most likely being added into the death count, we may see a duplication of the confusion between dying with or because of that we've seen with hospitalizations.  As it stands, with the current definition probably around 20-25% of covid deaths are with covid.  Many countries other than the U.S., such as the UK and Germany, also use the same reporting standards.(1)

In the near-term I think this issue with how we report the metrics will make it more difficult to determine what is actually happening on a daily basis and will add to public confusion.

In any event, if you're old folks like us, or have risky health conditions, get vaccinated, including your boosters.  Though we've discovered it will not prevent infection as we originally thought, it still greatly reduces your chances of bad outcomes.  Of that there is no doubt.

-------------------------------------------------------------------------------------------------

(1) While there is some uncertainty in calculating covid deaths because of v with, I wanted make sure my views are clear.  Since early in the pandemic I've seen claims that most covid deaths are with, not because of covid.  This is false - I've never been able to confirm any such claims.  Also, whatever uncertainties there are about the covid metrics, it turns out that flu metrics (to which people minimizing the risks of covid like to make comparisons) are even more uncertain and, in my view, more likely to be too high than covid metrics as I discovered looking at the CDC flu methodology back in April 2020.