How many eyeballs to make a bug shallow?

In the The Cathedral and the Bazaar, Eric Raymond said that “Given enough eyeballs, all bugs are shallow.” Of course, what he meant is when you have a large enough tester base and lots of developers, almost all bugs are found, characterized (i.e., how to make them reproducible and what are the effects) and fixed by someone in the developer group. This appears to me as somewhat optimistic and begs the question… how many eyeballs does it take to make a bug shallow?

To answer this question, we must define a “bug complexity” scale. Very obviously, bugs like off-by-one errors would receive a very low score on the scale. Hard to reproduce bugs, such as race condition based bugs, should probably receive a higher score, even if many of them are relatively simple to find and fix. How we measure the difficulty of finding a bug depends on this scale. Let us not define it explicitly but rather by its properties. Consider the following graph:

The Eyeball to Bug Complexity Function

The Eyeball to Bug Complexity Function

If you are an optimist, you will chose the dot-dashed dark blue curve as your “eyeballs to bug complexity” function. According to this function, a very complex bug can be found and fixed by a relatively small number of persons. This basically states that no bug is really complex.

The green line is more realistic, as it states that it takes a number of people linearly proportional to the complexity of the bug. Very complex bugs require a lot of people; but simple bugs can be fixed by one eyeball.

The dashed pale blue curve is an optimist/pessimist curve as it hints that simple enough bugs can be found and fixed with eyes shut, literally. But it states also that as bug complexity grows, so does the number of needed eyeballs. Complex problems will require a large number of people to be fixed.

The red curve is definitively pessimistic, but, in my opinion, the most realistic. Even simple bugs requires the two eyes of a single programmer, and the number of eyeballs grows rapidly in the complexity of the bug. Very complex bugs therefore need an inordinate amount of people to be found and fixed.

The fact that programming is inherently hard should not be news to you, but it is even more so than most of us actually imagine. Every one of us has a limited amount of intelligence, even the smartest ones, and that intelligence puts limits on what we can draw and understand from the world. Something might not be a bug if you don’t even understand what it should be doing in the first place. And even if you find a conspicuous bug, you still have to figure out how to fix it so that the program does what it needs to do, on a larger scale. Maybe, after all, all bug aren’t shallow. Even relatively simple bugs, like the 25 years old BSD bug, may elude the scrutiny of many expert eyeballs for a very long time.

(c) Philippe Geluck

(c) Philippe Geluck

4 Responses to How many eyeballs to make a bug shallow?

  1. Ex Girlfiend says:

    The topic is quite trendy on the Internet at the moment. What do you pay the most attention to when choosing what to write about?

  2. Steven Pigeon says:

    Daily activities as a computer scientist give me plenty of things to think and write about. Sometimes two different trains of thought collide to form a new topic, as it was the case here.

  3. Ross C says:

    Very good article. I always had the feeling there was something wrong with Raymond’s “eyeballs/bugs” theory, but I couldn’t put my finger on it. I think you’ve got it exactly right.

  4. […] a while because not all bugs are shallow and you don’t necessarily have an infinite amount of eyeballs at your disposal. Because some guy neglected to test his code thoroughly, you waste precious time […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: