The so-called Motte-and-bailey castle is a fortification consisting of a wood (most commonly) or stone structure built on a raised area of ground (the motte), accompanied by a walled courtyard (the bailey) and surrounded by a protective ditch and palisade. They were first built in the 10th century and were first introduced to England and Wales following the Norman invasion of 1066. Most of the castles that were built in the Middle Ages have not survived to the present day. Only the ones made out of stone have survived. As such, when we think of castles today, we almost always think of the ones built out of stone. It's a cognitive bias known as survivorship bias, a form of selection bias in which we tend to concentrate on the people or things that made it past some selection process (e.g. the stone castles) and overlook those that did not (e.g. the wooden castles).
There's another well-known example of survivorship bias from World War II. Abraham Wald was a mathematician working at Columbia University during World War II. Wald and his team of mathematicians were trying to determine how best to prevent U.S. airplanes from being shot down. Engineers could add steel armor plates to the airplanes to protect the pilot and crew and other vital areas of the plane, but the problem was that steel armor plates also made the airplanes heavier and slower to fly. Wald's team was tasked with figuring out where to strategically place the armor plates so as to keep the planes from getting shot down without sacrificing speed, which clearly was also important.
Wald's team examined the damage done to all of the airplanes that returned at the end of a mission. These airplanes were often riddled with bullet holes, so Wald's team determined where the bullet holes were concentrated the most. Here is what they found (the plane's location is on the left, the concentration of bullet holes, in terms of the number of holes per square foot, is on the right):
Engine 1.11
Fuselage 1.73
Fuel system 1.55
Rest of the plane 1.80
The team reviewed data that looked something like this:
At first glance, the military engineers recognized an opportunity to maximize protection without sacrificing speed, thinking that they could get the same protection with less armor if the armor was concentrated on the places with the greatest need, i.e. where the planes were getting hit the most. Abraham Wald disagreed. He told the military engineers that the armor plates don't go where the bullets are but where they are not. He recognized that the military engineers were making a mistake due to survivorship bias. Rather than focusing on the actual bullet holes, Wald suggested that it was the missing bullet holes that were most important.
It didn't make sense to Wald that the bullet holes would be concentrated in different parts of the plane. They should be distributed evenly. So where were the missing bullet holes (the ones that should have been on the engine casing if the damage had been spread equally all over the plane)? Wald suggested that the missing bullet holes were on the missing planes - the ones that didn't make it back. The reason planes were coming back with fewer hits to the engine is that those were the planes that didn't make it back. The large number of planes returning to base with most of the bullet holes concentrated on the fuselage was really solid evidence that hits to the fuselage weren't likely to cause the plane to crash. Hits to the fuselage could therefore be tolerated. In other words, the armor plates go where the bullet holes are not - on the engine casing!
There are many more examples of survivorship bias. We should recognize and remember that the survivorship bias is a statistical artifact - in other words, we should ignore it. The next time someone tries to imply that a correlation between an observation and an outcome implies causality, we should look for evidence of the survivorship bias. Chances are that we will find it. And then we will remember about castles and bullet holes.
No comments:
Post a Comment