In the The Cathedral and the Bazaar, Eric Raymond said that “Given enough eyeballs, all bugs are shallow.” Of course, what he meant is when you have a large enough tester base and lots of developers, almost all bugs are found, characterized (i.e., how to make them reproducible and what are the effects) and fixed by someone in the developer group. This appears to me as somewhat optimistic and begs the question… how many eyeballs does it take to make a bug shallow?
To answer this question, we must define a “bug complexity” scale. Very obviously, bugs like off-by-one errors would receive a very low score on the scale. Hard to reproduce bugs, such as race condition based bugs, should probably receive a higher score, even if many of them are relatively simple to find and fix. How we measure the difficulty of finding a bug depends on this scale. Let us not define it explicitly but rather by its properties. Consider the following graph:
If you are an optimist, you will chose the dot-dashed dark blue curve as your “eyeballs to bug complexity” function. According to this function, a very complex bug can be found and fixed by a relatively small number of persons. This basically states that no bug is really complex.
The green line is more realistic, as it states that it takes a number of people linearly proportional to the complexity of the bug. Very complex bugs require a lot of people; but simple bugs can be fixed by one eyeball.
The dashed pale blue curve is an optimist/pessimist curve as it hints that simple enough bugs can be found and fixed with eyes shut, literally. But it states also that as bug complexity grows, so does the number of needed eyeballs. Complex problems will require a large number of people to be fixed.
The red curve is definitively pessimistic, but, in my opinion, the most realistic. Even simple bugs requires the two eyes of a single programmer, and the number of eyeballs grows rapidly in the complexity of the bug. Very complex bugs therefore need an inordinate amount of people to be found and fixed.
The fact that programming is inherently hard should not be news to you, but it is even more so than most of us actually imagine. Every one of us has a limited amount of intelligence, even the smartest ones, and that intelligence puts limits on what we can draw and understand from the world. Something might not be a bug if you don’t even understand what it should be doing in the first place. And even if you find a conspicuous bug, you still have to figure out how to fix it so that the program does what it needs to do, on a larger scale. Maybe, after all, all bug aren’t shallow. Even relatively simple bugs, like the 25 years old BSD bug, may elude the scrutiny of many expert eyeballs for a very long time.