"100 apples divided by 15 red herrings"

I have posted about the need for public reporting of quality metrics in the past (see my post, "In God we trust - All others bring data").  In order to fully understand any quality metric, however, we have to be fully transparent and consistent on how we define these metrics.  With that in mind, I was particularly interested in an article published several years ago by Dr. Lisa Iezzoni ("100 Apples Divided by 15 Red Herrings: A Cautionary Tale from the Mid-19th Century on Comparing Hospital Mortality Rates").  Dr Iezzoni stated very eloquently, "Even hospital mortality rates, a seemingly straightforward staple of many provider 'report cards', can be modulated by the manner in which they are calculated."  

Dr. Iezzoni told the story of Florence Nightingale, a 19th century British social reformer and statistician who is perhaps best known as the founder of the nursing profession.  Following her experiences as a nurse during the Crimean War, Nightingale published her Notes on Hospitals, in which she recommended a number of changes in how hospitals could be set up and operated to minimize the number of deaths caused by unsanitary conditions.  She turned to another prominent social reformer and physician, William Farr to collaborate (a very early example of the importance of dyad leadership in health care).  Both Nightingale and Farr had significant concerns about the number of deaths occuring in English hospitals, particularly urban hospitals (at the time, mortality rates were believed to be higher in urban hospitals compared to rural hospitals, though this may have been related more to case-mix - see below).  

Nightingale and Farr conducted an analysis of the number of deaths occuring in the "106 principal hospitals of England" in 1861 and reported a mortality rate of 90.84% in the 24 hospitals in the city of London!  They also found a mortality rate of 39.41% in 25 county and provincial hospitals, so on the face of it, their analysis appeared to support the hypothesis that patients were more likely to die in urban versus rural hospitals.  Even more impressive in my mind was the mortality rate reported by the 13 military hospitals (15.67%)!

Let's take a closer look though.  Mortality rates were calculated by dividing the total number of deaths at the hospitals in 1861 by the number of patients at the hospitals on April 8, 1861.  What?!?!?  In other words, the numerator was based upon the total number of deaths that occurred over the period of one year, while the denominator was based upon the number of patients in the hospital on a single, specific day.  That just does not make sense.  Using the total number of patients in the hospital over the entire year as a denominator, the mortality rate in the London hospitals dropped to 9.7%.

I don't think that any hospital today reports mortality rates like Nightingale and Farr did in their analysis.  However, the point here is that you need to be extremely careful when comparing mortality rates between hospitals.  Some experts even feel that comparing hospital mortality as a quality metric is inappropriate!  Back to the 1860's, many rural hospitals explicitly refused patients with tuberculosis or who were close to death, whereas urban hospitals admitted everyone.  In other words, case-mix by severity of illness also matters.  As another 19th century critic wrote, "Any comparison which ignores the difference between the apple-cheeked farm-laborers who seek relief at Stoke Pogis (probably for rheumatism and sore legs) and the wizzened, red-herring-like mechanics of Soho or Southwark, who comes from a London hospital, is fallacious."  Even today, hospitals can differ significantly when it comes to the kinds of patients that they take care of, and adjusting mortality for these differences is fraught with problems.

It's amazing to me that we are having the same discussions today that were going on in the 1860's!  The take-home point is that transparency about how metrics are defined is just as important as transparency about those metrics themselves.  We don't want to be in a situation where we are dividing 100 apples by 15 red herrings!