Rainbow Testing

The Art of Estimation

Main Entry: 1es·ti·mate  Pronunciation: es-tə-māt

Function: transitive verb; Etymology: Latin aestimatus, past participle of aestimare to value, estimate

Date: circa 1532

1archaic a: esteem b: appraise 2 a: to judge tentatively or approximately the value, worth, or significance of b: to determine roughly the size, extent, or nature of c: to produce a statement of the approximate cost of

3: judge, conclude

 Main Entry: 2es·ti·mate  Pronunciation: es-tə-mət

Function: noun

Date: 1552

1: the act of appraising or valuing : calculation 2: an opinion or judgment of the nature, character, or quality of a person or thing 3 a: a rough or approximate calculation b: a numerical value obtained from a statistical sample and assigned to a population parameter 4: a statement of the cost of work to be done

 Main Entry: es·ti·ma·tion 

Pronunciation: es-tə-mā-shən

Function: noun

Date: 14th century

It’s about judgment; appraisal; a rough, tentative approximation of value, best handled with calloused hands and a skilled eye. This is not the stuff of laboratories with their over sanitized exactitude. This is about the credibility of well worn boots and measured words. Estimation is an art based on the weight of experience and the credibility of reputation. This is where theoretical science marries the real world. It’s imperfect, it’s challenging, there are wins and losses, but the value gained makes the effort worthwhile.

How do we bring together theory and reality in a way that actually brings value to the project table?

It’s a necessity for planning, and for creating the space necessary to perform proper testing and reduce risk. Yet many testers have little experience in it. Some balk at the suggestion of it. There are those who feel greatly suspicious of it.

Most testing education focuses on uncovering the exact truth, providing statistics that reveal precisely what has happened.  Testers exist to validate, not to project. Having invested years of experience in proving the accuracy or inaccuracy of the predictions of others, testers may be rightly loath to engage in such a speculative activity as estimation.

But to gain expertise in this skill will bring to a project valuable planning information that no other team representative can provide and can go a long way in obtaining for testers the one quantity so frequently in short supply: enough time to test. It’s a gem worth panning for.

Let’s examine the known quantities that provide a basis for the exercise.

Perhaps you have no more information than the development timeline and the high level requirements. Is estimation impossible? No, you can still start from there.

Industry Standards

What is the project management standard for estimation? In the absence of all other information 40% of development time plus or minus the level of confidence in the estimate is a good starting point. Where did the magic 40% come from? I have not been able to find a clear scientific reference for this. (I’m open for comments that may point to that source!) However, it is a reliable project management “standard” so you won’t be faulted for using it.

Confidence Level

How sure are you that this is the right amount of time? This level of certainty is your confidence level. It’s not a random number but a number about randomness. Confidence levels are generally used as statistical measures of certainty in results. We can use this to express the randomness of the data we are working with. A confidence level of 50% indicates the estimate is truly random based on pure speculation, with about a 50-50 chance that the time estimated would be the actual outcome. At 75% the odds are still not good - there's a one in four chance that your estimate is meaningless. This describes a situation where only a small portion (less than 25%) of information is available for prediction. Some statisticians consider 90% to be the minimum confidence level for statistically significant results. This would be based on known, repeatable results. If you have historical data that directly relates to your estimate, you can increase your confidence level accordingly.

And in the absence of all other information you cannot really have better than a 50% level of confidence in the estimate. So you start with 40% of dev time, the industry standard, multiply and add half again. Or more if you really aren't sure.

(40% x Development time) = initial estimate

(initial estimate x 50% confidence level) + initial estimate = final estimate

In case you’re wondering, you can save time by combining the two formulas. Just multiply the development time by 1 2/3 or 166.6667.

Increasing Your Confidence: the Work Breakdown Structure

If the test team is in house, the development is standard and there are metrics for set up and test case development, for turnaround, for defect rates and time actually spent running the tests; then you gather all of that information and start revising your estimate.

And multiply by your level of confidence.

As you gather this information you can begin to set up a Work Breakdown Structure (WBS) using Excel. Record every activity that you identify as being a part of the testing process and enter all the data that you have about the time involved. Record the activities even if you lack historical metrics on them.

You now have a list of activities, each of which has an historical metric, or an estimate. Remember to buffer, but be honest about what you are using as your basis. It's better for the process to be transparent, it invites people to volunteer information they may have been keeping to themselves and it instills confidence in your method.

The Value of History
When much of your data is speculative both you and your audience will lack confidence in it. This leaves a lot of room for your test time to be cut back as the project experiences stress. Wherever you can insert real historical data into your equations you will increase the value of your result. If it has never taken less than a week to run a known suite of tests then it is not likely that it will take less than a week in your upcoming test schedule. If the suite must be run, then at least a week must be scheduled. Real gold doesn’t weigh any less in the mountains than it does at the surveyor’s office. Your consistent metrics are your investment for future project planning. If you’ve been putting off gathering them, you can begin now. The project that you are estimating will provide a new historical beginning.

Application in the Real World: Putting the Rubber to the Road
On one of my last assignments I was working with an entirely new test team (theirs!) and some new processes. I had no familiarity with their tests, their environment, or their application. I gave them some training and we immediately began working on a major development project. I needed information from them in order to support project management.

I worked with the new team so that they could help me. I showed them how to realistically measure the amount of work that they covered in a work day, and how to account for time out for meetings and real world interruptions. Testing is never actually accomplished by the minute, and rarely by the hour. In the end I had testers who were pretty savvy about providing information for estimates and I was able to provide estimates to project management: revised at every major milestone, tightened down by actual experience and buffered by confidence level.

It’s not really a science, it’s an art, but you already have more information and experience than you might have suspected. In the end, the more practice you give yourself at this speculation, the better you will become at finding the gold: the right amount of time for the testing you need.

References:

www.m-w.com

http://www.malwarwick.com/learning-resources/confidence-level-calculator.html

40% is a Standard?

{ 07:00, 18 February 2007 } { Posted by strazzerj }

I've never heard of 40% as a "Standard", nor have I ever created an estimate based on a 40% starting point.

I suspect it may be folklore at your company, or it may be a magic number that has worked well historically at your company.

If one of my QAers gave me an estimate of 40% and justified it by saying it was some sort of "Standard", that wouldn't fly.

I agree that people need some sort of guideline to use "in absence of any other evidence", but 40% would not be an appropriate starting point for everyone.

40%

{ 09:00, 20 February 2007 } { Posted by prainbow }

Well, thank you for taking the time to look this over. I appreciate it.

Honestly, I have heard the 40% at a couple of companies as a starting point, but have never been able to track down a source. As an absolute starting point, in the absence of other information, what would you use? I'm always interested in this because I would love to have a counter point to the 40% figure.

Paulie

It depends ... (of course)

{ 05:17, 20 February 2007 } { Posted by strazzerj }

I try very hard not to give any estimates without knowing any information.

Still, where I work now, when forced to do so we use 50% and work up or down from there.

At a different place I have worked (with different circumstances) we used 100% of development time as a starting point.

I guess you have to start somewhere!