I've been thinking a lot about measurement lately. I recently attended the 2024 Critical Care Congress of the Society of Critical Care Medicine, where I gave a presentation entitled "How to Use ICU Metrics and Financial Data" at the Leadership, Empowerment, and Development (LEAD) Program. Later during Congress, I participated in a Pro/Con style debate, where I gave a presentation entitled "ICU Metrics Don't Improve Outcomes" (I was the "Con"). Thankfully, both presentations seem well-received. Here are the main points from both talks:
Most commonly used health care quality metrics actually don't measure the quality of care.
I've made this point a number of times before in previous posts (see "The cost(s) of quality", "You give to get to give", and "Invest your money wisely..."), and I again referenced an important study published by Dr. Elizabeth Bradley, a health policy researcher and President of Vassar College, who also wrote an excellent book on this subject called The American Health Care Paradox: Why Spending More is Getting Us Less. Dr. Bradley makes the compelling case that if we truly want to improve health (as measured by the commonly cited quality metrics of life expectancy, infant mortality, maternal mortality, etc), we should be investing our money on improving the social determinants of health. We often hear that despite spending more than any other country on health care ("care" being defined as what clinicians do for their patients in the clinic or hospital setting), health outcomes in the United States are are among the lowest for developed countries. Unfortunately, the quality of the American health care delivery system is measured by outcome metrics that are mostly determined by what happens outside the care delivery system (some say as high as 80% of outcome metrics are determined by the social influencers of health). We are measuring health delivery and not health care delivery. My point here is that we should use the correct set of metrics that are measuring what we are trying to improve.
Process measures don't always accurately reflect outcomes.
Avedis Donabedian, a physician and one of the founders of the health care quality movement, described what is now called the Donabedian model (see the Figure below). Donabedian believed that the quality of care can and should be measured using structure, process, and outcome metrics. Structure refers to all of the factors that describe the context in which care is provided (i.e., physical facilities, staffing models, use of an electronic health record, availability of equipment, etc). Process refers to how care is provided (i.e., compliance with bundles). Outcome refers to the end-result of the care that is provided (i.e., length of stay, mortality, etc).
Several years ago, I provided a list of examples of commonly used ICU metrics, breaking them down using both the Donabedian model and the Institute of Medicine's six domains of quality:
The important thing to remember is that there is not always a cause-and-effect relationship between, in particular, process measures and outcome measures. Health care organizations typically measure process measures, as they are usually more easily measured. Unfortunately, process measures cannot always be used as a proxy for outcomes. Ideally, organizations should utilize a portfolio of measures that use all three - structure, process, and outcome.
Goldilocks was right.
I was visiting a health care organization a few years ago, and my tour guide was very excited to show me their "Improvement Wall". I walked into a small room and virtually every inch of the four walls were covered with charts and graphs of all the measures that they were tracking. Rather than being impressed though, I respectfully asked how long it took someone to print these charts and graphs out every week on a regular basis. The lesson here is that it is easy to get carried away, and by doing so, organizations will get lost in their measures. The "Goldilocks Principle" applies here - remember, the K in KPI stands for "key" performance measures. It's easier to "game the system" with too few measures, and too narrow of a focus will create blind spots. Conversely, too many measures leads to a loss of focus! We should be using a relatively limited set of measures that can be used to drive improvement. Consider what the Institute for Healthcare Improvement's President Emeritus and Senior Fellow and former CMS Administrator (and pediatrician), Don Berwick says about metrics. Berwick is a HUGE proponent of using data to drive improvement, but he lists "Metrics Glut" as one of his "Seven roadblocks to improving patient safety". He says, "We need to stop excessive measurement. I vote for a 50 percent reduction in all metrics currently being used."
Beware Goodhart's Law.
I've also posted a lot on "Goodhart's Law" (see "Your quality measure is no longer useful...", "Forced ranking - Goodhart's Law redux?", "Tyranny of metrics", and "You manage what you measure") that states, "When a measure becomes a target, it ceases to be a good measure." The economist, Charles Goodhart, first described the concept in 1975, stating more technically that "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." For example, if you measure productivity (and more importantly, incentivize) in a nail factory based upon the number of nails made, you may find that your factory produces thousands and thousands of very tiny nails that aren't actually useful. On the other hand, if you base productivity on the total weight of all nails made, you may find that your factory produces a fewer number of large, thick, heavy nails that also can't be used. There are a number of examples of Goodhart's Law in health care today - my favorite (discussed in my post, "Tyranny of metrics" which are also discussed in the great book by Jerry Muller) is the National Health Service's efforts to reduce boarding in the emergency department (ED) by instituting a 4 hour benchmark to evaluate, treat, and make a decision to admit or discharge a patient. When the data was graphed out (see the excellent review published by Julie Eatock, Matthew Cooke, and Terry P. Young), there was a peak of hospital admissions from the ED right at four hours! I suspect that some (not all) patients were admitted to the hospital rather than waiting a little longer to determine if they truly needed to be admitted, which is a waste of limited resources.
Beware the Cobra effect.
The "Cobra effect" (which I first discussed in "Is this another April Fool's joke?") is basically another name for something that is called the "Law of Unintended Consequences" (it's also another version of "Goodhart's Law"). As the story goes, during the British colonial rule of India, there was a big problem with cobras - they were killing a lot of people. The British government decided to control the cobra population (smart administrators that they were) by offering a bounty for each dead cobra that was brought to the local authorities. What a great way to encourage the local population to take matters into their own hands, right? Well, unfortunately, the plan backfired. The local entrepreneurs started breeding cobras so that they could turn more cobras in to the government authorities and make a nice profit! The government caught on fairly quickly and abandoned the program. The entrepreneurs released the cobras that were bred in captivity into the wild - and the cobra population increased to even higher numbers than before the program started! Similar to the point I made above, be careful of what metrics and what benchmarks that you incentivize, as there are always unintended consequences.
Measure what matters.
Building upon the "Goldilocks Principle" above, W. Edwards Deming, widely considered one of the founders of the quality improvement movement, listed 14 points for quality management. Point #11 (there were actually two, which he labeled 11a and 11b) is:
11a. Eliminate work standards (quotas) on the factory floor. Substitute leadership.
11b. Eliminate management by objective. Eliminate management by numbers, numerical goals. Substitute leadership.
He also listed the 7 deadly diseases of management, of which #5 is Management by use only of visible figures, with little or no consideration of figures that are unknown or unknowable. What does it say when two of the leading authorities on quality improvement in history (Deming and Berwick) caution against excessive measurement and key performance metrics? While it is certainly true that you can only manage what gets measured (to paraphrase Peter Drucker), you should only measure what truly matters (see the superb book by John Doerr, Measure What Matters) and needs to be managed!
Accountability and authority must align.
I've also made this point before (see my Accountability-Authority Matrix), but don't just listen to me. Joseph Juran, one of the other founders of quality improvement, states that when it comes to managing quality: (1) Goals must be clearly defined; (2) Actual performance must be measured; and (3) Authority to act must be present. Individuals need the authority to be able to act on data to drive improvement. In addition, they should not be held accountable for metrics that they have no authority to influence or change.
Please don't misunderstand me. I am a firm believer in measurement. It's one of the foundational principles of quality improvement. However, as in all things, there are a few important caveats that should be kept top of mind, some of which I have listed here.
No comments:
Post a Comment