Tyranny of metrics

I came across a cartoon the other day (see here) that reminded me of a visit I once made to another health care organization.  One of the hospital executives was showing our team around their C-suite, and we came to an area where they proudly displayed all of their Key Performance Indicators (KPIs).  The individual was particularly proud of the fact that the wall (and it was a large wall) was covered from top to bottom with charts showing several (and I do mean several) KPIs graphed over time.  Each KPI was displayed in color on a run chart using an 8.5 x 11 inch piece of paper.  I couldn't help but ask two questions, "Who collects all of this data and how much time do they spend each week on updating all of these charts?"  

We've unfortunately come to a point where a book by the author, Jerry Z. Muller called, appropriately enough, The Tyranny of Metrics should be required reading!  Please don't misunderstand me here, I am a big believer in having data available to quantify - objectively - either your own performance or the performance of your organization.  I loved the book by John Doerr called Measure What Matters, and I loved what W. Edwards Deming had to say on this subject when he said, "In God we trust, all others must bring data."  Finally, I have always subscribed to Peter Drucker's theory that "If you can't measure it, you can't improve it."  However (and it's a BIG however), there is no question that some leaders and organizations have gone way too far when it comes to KPIs and metrics.

But please don't listen to me.  Take what the Institute for Healthcare Improvement's President Emeritus and Senior Fellow and former CMS Administrator (and pediatrician!), Don Berwick says about metrics.  Berwick is a HUGE proponent of using data to drive improvement, but he lists "Metrics Glut" as one of his "Seven roadblocks to improving patient safety".  Say what?  Too much measurement can actually prevent us from improving?  He says, "We need to stop excessive measurement.  I vote for a 50 percent reduction in all metrics currently being used."  Mark Graban, an expert in health care improvement and author of a great blog called Lean Blog, writes of Berwick's point here, "I agree that most organizations' executives have too many metrics - not just around patient safety and not just in healthcare.  Remember, KPI stands for KEY Performance Indicators."

Berwick often refers to an example from the British National Health Service (NHS) when he talks about the problem of excessive measurement - the "tyranny of metrics."  Apparently, in the early 2000's, the NHS, trying to address the widespread problem of emergency department (ED) overcrowding, set a goal that all patients presenting to the ED should be seen, treated, and discharged (or admitted) within 4 hours of arrival.  Take a look at the following graph from a review of the NHS goal published by Julie Eatock, Matthew Cooke, and Terry P. Young:

What does this graph tell you?  Do you think it is a coincidence that the percentage of patients who are admitted to the hospital suddenly spikes right before the four-hour mark?  Almost everyone of us procrastinate and then scramble to complete our assigned task so we make a deadline.  I can imagine that these hospitals are doing the same thing, delaying admission for as long as possible and then scrambling to admit patients right before the deadline.  Here is a great example of one of the major drawbacks identified by both Jerry Muller in his book and Don Berwick in a number of speeches and presentations - gaming the system (there's even a Wikipedia page for that - see here)!  I have written a number of posts in the past on something known as Goodhart's Law (see also here and here), the concept that once performance is tied to a particular metric, the metric ceases to become useful.  The NHS four-hour goal is an excellent example of Goodhart's Law!

So, how do we avoid the "tyranny of metrics"?  While his focus was on more than just excessive measurement, Don Berwick listed the following "9 steps to end this era of greed and excessive measurement":

1. Stop excessive measurement

2. Abandon complex incentives

3. Decrease focus on finance

4. Avoid professional prerogative at the expense of the whole

5. Recommit to improvement science

6. Embrace transparency

7. Protect civility

8. Listen. Really listen

9. Reject greed

I think that some of these points are particularly relevant to the "tyranny of metrics," particularly #1, #2, #3, #5, and #6.  I talked about the IHI's concept of Whole System Measures in my last post ("How good are we really?").  What's important to realize here is that (1) there is no single measure that can adequately measure the overall performance of a health care organization (which is likely true for any organization, really) and (2) a short list of measures is preferable to a large one.  

Please do yourself a favor and read both The Tyranny of Metrics and Measure What Matters.  Tell me what you think!  Seth Godin, another blog writer, says, "Measuring is fabulous.  Unless you're busy measuring what's easy to measure as opposed to what's important."  I think everyone I mentioned today in this post would agree.  I certainly do.