Putting Relative Estimation to the Test

Added to Process

Traditional agile forecasting involves breaking down an entire backlog into stories, getting point estimates on each story, summing these and then dividing the sum by the team’s velocity to yield a projected completion date. This process is time-consuming, taxing on developers, creates waste and the results are questionable at best. It also doesn't work well across teams (i.e. doesn't scale). Because of these flaws, I wanted to test the underlying assumption that relative estimation yields better results than absolute and see if there might be a better way to forecast using the wisdom of crowds.

Terms

Absolute Estimation: A straight out guess

Relative Estimation: Comparison based guess

Wisdom of crowds: A theory that averaging the guesses from a large population yields a highly accurate estimate

Understanding Relative Estimation Theory

There is a great article on the Agile Alliance website that outlines what Relative Estimation and the theory behind it.

Hypothesis:

If the underlying assumption of relative estimation is true, then a given population should be able to estimate the number of jelly beans in a container with higher accuracy than straight out guessing (absolute estimation).

Conversely – should the absolute estimate be closer in accuracy than the relative estimate, then absolute estimation with the wisdom of crowds could be considered a better tool for forecasting than relative estimation.

Materials Used in this Experiment:

Three transparent Tupperware containers:
- a small one
- a medium one
- a large one

Jellybeans (enough to fill the three Tupperware containers)

A notepad or spreadsheet to hold a data table of 150+ rows and two columns in which we will record our results. Columns are titled Guess One – Absolute, and Guess Two – Relative.

A population of around 150 people. (Dunbar’s number was chosen for this experiment as it seemed a good starting point for a “crowd”.)

One person to collect data.

Procedure:

A population of 150 people were asked to guess how many jelly beans are in a container. (An absolute guess.) After making this guess, they are then shown two different sized containers and told the amount of jelly beans in these containers. Armed with this new information, the same person then takes a second guess against the original container i.e. a relative estimate. The data from the two guesses is collected, recorded and finally collated for average across the population.

Step by step process

  1. fill the smallest container completely with jelly beans
  2. count the jelly beans in the small container (you may need to empty and refill to do this)
  3. record the count of the small container (perhaps write it on the bottom of the container so you don’t lose it)
  4. do steps 1 through 3 again with the medium container
  5. fill the large container with jelly beans. Do NOT count the beans in this container. It’s important that you (the surveyor) do not know until all the data has been collected, what the jelly bean count is for the large container so you cannot anchor in any way. Data collection should be done double blind.
  6. Collect guesses: You may survey individuals one on one or in groups. If in a group, it is important to communicate to the group that when they are making a guess, they keep it to themselves so as to not anchor anyone else around them. The spiel to participants should go something like this:- “Your involvement as a survey participant in this experiment is short and should take no longer than 5 minutes, and it usually averages around 2-3 minutes. I am going to ask you to silently write two estimates on a piece of paper. The first will be numbered 1 and the second numbered 2. Show an example of what you expect the piece of paper to look like at the end with scribbles where the number estimate is to go. For example “#1 <your guess>” and below it “#2 <your guess>”. Your guess being an integer that represents how many jelly beans are in a container that I am going to show you shortly. At that point, I will ask you for your first guess. Silently put your guess on your paper and be careful not to announce or share it in any way for fear of influencing others. Also, be vigilant not to look at others if at all possible so as to avoid any anchoring. After the first guess, I will give you some more information and then ask for a second guess which you will write in the number two position. Once I collect the piece of paper from you with your guess, then your participation is complete. Again – please do not share any of your guesses or say them out loud during the process. Any containers I show you, you may not ask about what the volume is.”

Results (Data):

JellyBeans1 containing the guesses that were collected for this experiment. Column one is the first absolute estimate, and column two is the second relative estimate.

Population size: 154 people

Actual number of jelly beans in container: 1009 (counted at the end of the experiment)

Population Average of EstimatesMargin of Error as a Percentage
Relative Estimation136134.89%
Wisdom of Crowds - Absolute Estimation10544.46%

Theory: Absolute estimation with wisdom of crowds is more accurate than relative estimation

Absolute estimation was roughly seven times more accurate than relative estimation (with a population size of 150) i.e. roughly 5% variance using absolute estimation compared to 35% (rounded) with relative estimation. This suggests that averaging an absolute guess from a large population is a better tool for estimation compared to relative estimation.

In this initial experiment, we chose a population size of around 150 based on Dunbar’s number as the maximum size of a tribe. A tribe sized group sounded like a good place to start for a crowd.

How big is a crowd?

Well, it depends. It depends on how accurate an estimate you want/need. Using the data of 154 participants, we pulled off multiple subpopulations of different sizes and plotted the %deviation against the population size. The graph will help you determine what size population you need, to what level of accuracy you desire. Note the diminishing returns as it the curve plateaus approaching 150.A rough translation of this graph shows that if I was happy with 80% accuracy, I could get that with a population of around 20-30 people. To get that to 90%, I will need around 60 people. 95% kicks in around 120 people.

Try it Yourself

If you have repeated this experiment for yourself or have done any further research into this field, I would be very interested to hear about it. Please email Ron Quartel using email address: info at fast-agile dot com. Watch this space for further experimentation that we are currently conducting into the area of estimation and forecasting.

Further Reading and References

Relative Estimation definition and history

The Wisdom of Crowds by James Surowiecki (Affiliate)

Blink Estimation

Dunbar’s Number

Another Example of the Wisdom of Crowds for estimation (Note that result was also within 5%) 

Sir Francis Galton (The founder of the wisdom of crowds theory)

About the Author

Ron Quartel is the founder of FAST Agile (http://fast-agile.com) and FAST Forecasting. FAST Agile is an experiment in using Open Space Technology and Code Craftsmanship for cross-team collaboration in large scaled Agile projects. In his day job, Ron is an Extreme Programming Coach (XP Coach) and Agile software developer. Ron describes his calling as “creating harmony between business and development”. Read Ron’s twitter feed under the handle @agileAgitator and blog articles at http://blog.agileagitator.com.


This is an Agile Alliance community blog post. Opinions represented are personal and belong solely to the author. They do not represent opinion or policy of Agile Alliance.