How do we define "reliability"?

I wanted to build upon the theme from last month on High Reliability Organizations.  How do we know when an organization may be legitimately called "highly reliable"?  Is there an objective measure of performance that organizations must achieve to become classified as High Reliability Organizations?  By extension then, can these same organizations "lose" their classification as a High Reliability Organization if they experience a catastrophic accident?  Alternatively, does the term High Reliability Organization represent some theoretically possible but realistically unattainable performance that organizations can only aspire to achieve?  As the legendary Hall of Fame Green Bay Packers coach Vince Lombardi, once said "Perfection is not attainable, but if we chase perfection we can catch excellence."

It seems like a good time to quickly go back and re-visit the history of the concept of a highly reliable organization, which was first described by an eclectic group of interdisciplinary researchers from the University of California, Berkley in 1984.  Todd LaPorte (a political scientist), Gene Rochlin (a physicist-turned-political scientist), and Karlene Roberts (a business school professor with an expertise in organization management) analyzed three vastly different organizations - the US Navy nuclear-powered aircraft carrier USS Carl Vinson (in partnership with retired Rear Admiral Tom Mercer), the Federal Aviation Administration's Air Traffic Control system, and Pacific Gas and Electric's nuclear power plant at Diablo Canyon.  Importantly, these three organizations were selected out of convenience, not because of any randomization or pre-defined criteria (basically, they used what is called a "convenience sample").  These investigators identified a number of key characteristics that the three prototypical HRO's shared, which included "engaging in considerable delegation of decision making and responsibility; structuring themselves to quickly move from completely centralized decision making and hierarchy during periods of relative calm, to completely decentralized and flat decision structures during 'hot times'; engaging in constant training and developing strong cultures of safety orientation and simultaneously embracing a complicated mix of very old and very new technologies according to what is most appropriate to the task at hand" (see "Learning how organizations mitigate risk" in the Journal of Contingencies and Crisis Management).

Importantly, most descriptions of so-called High Reliability Organizations fail to provide a definition of reliability in exact terms, i.e. the degree of reliability that is required in order for an organization to be classified as highly reliable.  For example, one article suggested that High Reliability Organizations could be identified by asking the following question: "How many times could this organization have failed, resulting in catastrophic consequences, that it did not?"  If the answer to the question was on the order of tens of thousands of times, then that organization could be classified as highly reliable (see "Some characteristics of one type of high reliability organisation" in the journal Organization Science).  While that definition makes sense, it is far from satisfactory.  As a matter of fact, Gene Rochlin himself said in 1993 (see "Defining high reliability organisations in practice: A taxonomic prologue" in Karlene Roberts' book, New Challenges to Understanding Organisations) that "no truly objective measure is possible" and "what distinguishes reliability enhancing organizations is not their absolute error or accident rate, but their effective management of innately risky technologies..."  Again, not very satisfying.  In the end, we are left with definitions that are both highly subjective and mostly descriptive in nature (see a great article by Andrew Hopkins "The Problem of Defining High Reliability Organisations").  

While I hesitate to add to the confusion here, I do think that it's important to quickly review some of the commonly used definitions of reliability (importantly - not necessarily high reliability).  For example, an error (or defect) rate is calculated by the number of errors that occur in a certain number of opportunities, typically expressed as the number of errors in 1,000 opportunities (or in some cases, per 1,000,000 opportunities).  Error rates can also be expressed as a percentage.  Finally, one commonly used method is to express the error rate on a logarithmic scale (10 raised to the negative power of X, where X denotes the denominator of the equation for error rates), as shown in the Table below.

For example, central line infections in a hospital's Intensive Care Unit (ICU) are often expressed as the number of infections per 1,000 central line days (where the denominator is the number of opportunities for infection due to the presence of a central line).  Most ICU's have central line infection rates less than 1 infection per 1,000 line days, which would equate to Level 3 reliability (10 raised to the -3 power).  Notably, most airlines operate at around this same level of reliability when it comes to baggage handling (i.e., they lose 1 bag per 1,000 opportunities).  

I recently came across the mathematician Terrence Tao's system of classifying reliability based upon a concept that he calls the "Nines of Safety".  For example, if a process is reliable 90% of the time, Tao would say that the process has "one nine of safety".  Similarly, if a process is 99.9% reliable, Tao would say that the process has "three nines of safety" (which equates to Level 3 Reliability discussed above).  It's an interesting way of expressing reliability.

Note that I am avoiding a discussion on Six Sigma reliability (a topic for another, future post), which is similar yet distinct to the methods described above.  I think that all of these ways to classify reliability can be helpful, and I suspect that most so-called High Reliability Organizations operate at a reliability level of 4 or greater (1 error per 10,000 - 1,000,000 opportunities).  David Ikkersheim and Marc Berg have suggested (see "How reliable is your hospital?" in the journal BMJ Quality and Safety) that most hospitals operate at a reliability level of 1 or 2, which is consistent with my own experience and observations.  In other words, most hospitals are not at a point where we would consider them to be High Reliability Organizations.  Regardless, high reliability is certainly an important and attainable goal, so the journey to become a High Reliability Organization continues.