Rainbow Testing

Regression testing � the Ugly Betty of Testing

I was once asked by a high level manager whether failing a regression test meant that there was regression, or there was not. Clearly we have failed to communicate.

Of course, we are checking to ensure that there has not been regression.

No regression, no backsliding, retrogression, relapse nor degeneration! We move forward without losing any ground.

Or so it is proposed.

Regression testing is the cousin to automation, the dowdy step sister of all the ***ier techniques applied to new code (which is the rich vein of imperfection that temps avid testers). Not so for regression testing. Few of the ambitious long for regression testing.

It is the stalwart helpmate of version control.

Regression testing, in an optimal environment, will uncover few defects, few mysteries. Its foremost weapon is so dull it is called a Smoke Test. It is so predictable that its secondary use is as a training tool for new testers. It presupposes a thorough and unshakeable knowledge of the base of the system under test.

It is boring.

Besides this it still requires constant maintenance. Little surprise then that it can be difficult to charm your best testers into taking it on. Testers generally lust after breakage the way the Cookie Monster craves chocolate chips. Regression testing at it�s best provides little of that type of excitement.

Then what is exciting about Regression Testing?

Regression Testing is a true thrill to:

� anyone who is stuck in a CMM level 1 environment and knows in their heart that it doesn�t have to be like that;

� any tester who had the experience of dropping an hour�s worth of work into fresh code only to find out that the whole thing is invalid because the code drop had a fatal flaw;

� anyone who has had to spend hours refreshing test data that got burned off against a bad version

� any cross over automation testing geek who gets a charge out of setting the stage for lighting fast diagnostics

You get the picture. Regression Testing appeals to the folks who are drawn to testing because they like straightening things up and sorting things out.

Some testers like seeing the defect count. Others like taming the Wild West of the Software Development Lifecycle.

If you decided to take on the Regression Testing then Ugly Betty just bought herself a saloon.

Your six guns are going to be those gun metal grey smoke tests. Oil them up. You are looking for fast, brutal paths through the system. No depth here, just run that bullet through every ten gallon hat you have lined up easy, without any fuss.

If this is an order system, you want to have a test that runs an order, end to end, using the most typical choices possible. Fall in love with default processing for this one. You want an answer to this question in 3 notes or less: Is the system testable?

If you cannot get the simplest order through the system, the simplest user path completed end to end, stop the piano player and throw that buckaroo out of the bar. Stopping a bad build with a good smoke test is as easy as licking butter off of a knife. Congratulations, Regression Tester! You just saved your team and your company hundreds of dollars in wasted time.

What�s next?

If the system passes the smoke tests you can, if that�s your process, hand it over to the eager testers that forge their way across new lands in search of problems and unmet requirements, but you�ve still got work to do, and the faster the better.

Remember that the goal of testing is to find the most significant problems soonest. Leave it to the rest of the team to be vetting the highest level requirements etc for the new development for you this means something different, and possibly something undefined.

What�s the most significant problem?

Your research for your Regression Suite should have included a chat with some of the good people in Finance. What�s the company�s money maker? It needs to be tested. Find someone who can run you reports on transaction distribution. What is the user doing the most? The point of Regression Testing is to validate that what has been happening can still happen. Put the things that happen the soonest and the most often at the top of the list.

It�s inevitable in software development; the user will be disappointed somehow, some day. It�s best if that�s not their first experience, or even in their first 5 minutes on the new code.

Tight Scrouging

Here�s where you get to hoe the corn. This is the real test of what you know, or what you can find out, about the end to end system. A first class regression test is going to have a case set up to hit nearly every single system end to end. Again, no depth, but in this case, or these cases (you may need more than one for coverage) the object is breadth.

Most systems have certain exception criteria that will send the user path off into a specific function or subsystem based on certain specific criteria. The real ace for this set of cases is an input that will send the user through as many of those as possible.

Use a spreadsheet, use a mentor, work with a team, but make sure you have a set of tests that hit all of the functions and subsystems with the minimum amount of input and individual cases.

Remember those scenes in your favorite old westerns where the sharp shooter looks in a mirror, aims over their shoulder and bounces a bullet off of every hard surface in the saloon before blowing the hat off of the skeptic? You are that sharp shooter. Go ahead and blow their hats off.

Recording and reporting

You are going to have to change your perfect test. The system will change and grow and you are going to have to move with it. Make sure that you have classified your test anatomy properly. Be clear about your inputs, your test data. If it�s easy to find it will be easy to update� and easy to automate. In fact, take some time and talk to the automation tester that you are working with. You�ll both save a lot of time and money by having overlapping approaches.

Good coders leave clear extensive comments that explain what they were intending when they wrote the code. Don�t you be shy about leaving clues for the history books. Make sure that your test format allows for commenting. If the case relates to a specific requirement or old defect, record that.

Make sure that you can sort out the test cases that target specific functions, inputs or subsystems. This allows you to run subsets of the larger test suite or make quick updates across the Regression Test when a specific area changes.

How does the Configuration Management (CM) system classify the parts and pieces of the system? Does your system dovetail with that?

What metrics need to be collected from testing? Does your Regression Test provide input for that?

These are the kind of questions that can help you double check your Regression Test strategy against other systems in use in the enterprise.

In the end, while the denizens of your saloon have their own high adventure, they probably won�t give a thought to the work that goes into keeping the environment open for that fun. That is as it should be.

Let the adventure be the new code, not the same old chaos. You�ve now got a tight grip on a bucking bronco and that�s something to be proud of in this Wild West. Forget Ugly Betty, you are the Money Making Mama of this here town.

�Greater is the art of ending

As pointed out by Lazarus Long, it is ending that requires a certain skill. There are miles of words written about getting positions but few written about the successful leaving.

Why end well?

Other than the simple exchange of work for money are three main benefits of your work contract. These are:

            The accomplishment of your client�s goals
            Work experience
            Professional contacts

Ending well ensures that these valuables are preserved.

So, whether you expected it or not, you�ve been informed that your contract is at an end or that it will not be extended. Good. You know where you stand.

Pull up the initial Statement of Work, or the earliest communications about this position. This might be the first email your recruiter sent you with the details of the position. Pull out the meat of it, make a separate list if you have to. Can you tick off what was actually accomplished?

Make a similar review of any outstanding task list. How have you been tracking the work? A project tracking program? A weekly status report? Email? Sticky notes? It�s time to round up the details.

Circle the loose ends. Did your client change their mind? Was there an external change that pulled certain items off of the list? Would you and your client have the same point of view on this?

Look at the real loose ends, the items that you were set out to accomplish but were not completed. Can any of these get done? What can you do to support you client in having them finished after you leave? Do you need to leave documentation? Do you need to set up a meeting with someone who will be picking up the task when you are gone?

Now sit down with your client contact. Do this as soon as you can. Review what you know, briefly. Make sure that you and your client are on the same page where your last billable hours are concerned. Most clients will have some clear ideas about what you should be doing with them, but there may be details that they have overlooked or valuable suggestions you can make to help them make the most of your work after you have gone.

Naturally you will complete your work and report on your status.

But you have the opportunity for a little something more.

Put together a single page summary of your contract accomplishments, one that is paired with the Statement of Work if possible. This will provide your client with valuable information of what they have gained through the contract.

It also provides you with a valuable list of your own experience. Keep a copy and on your copy remember to note the systems and applications that were involved. Make sure these match your updated resume.

You have probably been gathering contact information all along. If not, now is the time to ask your work contacts if they can be your professional contacts. Make sure that they have a business card, a v-card or are willing to be your professional contact on a business networking site like LinkedIn. Even if they do not wish to be a professional contact for you, make sure that you have the right people listed privately so that you can remember their names and send thank-you notes.

Yes, thank-you notes.

You�ve been reminded to send them to interviewers and prospective clients, they are equally important in the art of ending. Take the time to send them out the week after you leave.

And now you will have wrapped up the valuables of your time there. You have a solid relationship with your past client. You have a list of your recent accomplishments. You have a precious handful of professional contacts.

And you are ready to start again.

Zombie projects � when the software lifecycle won�t die

Your original planned installation date was 6 months ago. Management has repeatedly lumped and un-lumped your project in with other various releases, or split the original project into various phases, sometimes in ways that defy logic.

From each phase, defects that didn�t create too much impact but would be challenging to fix were deferred to the next phase, and then the next.

Now most of the functionality is in production, primary resources have been moved off the team, current deadlines are vague, and there�s a batch of outstanding issues that have been pushed off for so long that you�re no longer entirely certain how to deal with them.

It�s the long, slow death of a project that has been in trouble for a while.

Step one, account for the living

First of all you should count your successes. If indeed most of the functionality has been released and is being used, count yourself lucky. Many, many projects never make it that far. If you take a moment and clearly enumerate just what has been accomplished; celebrate the living; it will give you the fortitude to face the undead.

Round up the zombies

They are strong, they are brutal, they are insensitive to pain, but zombies aren�t very agile. Use that most basic of QA tools, the list, to round them up so you know what you are dealing with. What do you know for sure about this project? What functionality is still being worked on? What releases have been agreed to? There are some tasks and defects that will fall under those two categories and those are the living, for now.

Use light and fire

What is unclear? Clarification is the light that zombies fear. List the items that have not been released, aren�t being worked on but have not been killed off. These are the things that are neither living nor dead. Your quest is to track down the stakeholder who can own them or remove them from the project. For every clear decision made: publish. Clear decisions are the fire of zombie destruction. Put them in broadcast emails, or weekly reports, or bring them up in status meetings and make sure they make it into the meeting notes. These zombies are dead.

Defects: little zombie dust bunnies

Don�t overlook your defect list. Sure, you�ve got them rounded up and you might even have a person assigned to them, but like zombie dust bunnies they have a way of lying dormant, inviting you to overlook them and leave them to their quiet half-life, eating up space in your defect database and forcing you to get creative in your filters to push them out of the way in your reports. You can get rid of this innocent looking dead weight. Publish the obituary in the release notes under �known issues� and close the defects as �not a defect� or some other category that demonstrates their dead state. Make sure that you reference the release note version and date in your defect comments.

Zombie self defense: work together

Are there objections to leaving a defect unfixed? See if the objectors are willing to become stakeholders for the reanimated issue. Can you assign this to a specific future release? Can you hold a final defect decision meeting where the stakeholders make up or down decisions on each issue forcing the undead into the light of day? Remember, according to Tracy Wilson of howstuffworks.com, you should avoid the common mistake of letting personal feelings or arguments drive your zombie fighting team apart. (She also advises not to teach zombies how to use firearms.) It�s a messy business but you�ll all feel better when it�s done.

After the darkest night the sun will rise

Back at your own workstation run a last report and breath in the satisfaction that comes with a clean list of truly live issues that have assignments and release date expectations. Send out that project follow up email that declares the end of the zombie project and congratulates your fellows for the work accomplished. You�ll find yourself looking forward to the new start of the next phase or project, free and clear of the issues that haunted you before. Your project is dead. Long live the new project!

The test of true love

What is it about love that so blinds us to faults? It�s a power that makes it possible to adjust to the inconveniences and imperfections of our commitments, but can make us fatally vulnerable to the flaws that truly impede us on our path, that may block us even from every happiness and all the things we truly value.

This is no rhetorical question for a tester to consider but a real issue that can block us from our great asset; objectivity.

Loss of objectivity

There are numerous circumstances that can lead to a bias that affects planning, testing and reporting. Here are some common examples:

Financial investment
High level corporate decisions
Over confidence in the development team
Frustration with existing systems
Marketing
Lack of software development expertise
The overwhelming desirability of a single aspect or function of the proposed system
Over

Undoubtedly you could increase this list from your own personal and professional experience but they can be boiled down to two primary drives; a revulsion for something unwanted or a desire for something idealized.

Putting it back into perspective � if I could write a sonnet�

Don�t be afraid to �count the ways.� If you, your management or your user group are caught up in the hype, write it down. List clearly the benefits that are driving the excitement. Most testers can spot ambiguous language, indefinite pronouns or unclear clauses even in a love letter. Fall back on your talents. Write it out as a list. Then take out that valentine red pen and go to work, doing what you do best, question the language. Any item that cannot be clarified is a risk.

Avoiding the reactionary � the enemy of my friend is�.

Read the criticism, listen to the naysayer�s, make a note of the Chicken Little�s. Once again, rely on your innate skills. Pick up any single article or complaint and make a cuff list of the pertinent points. What are the primary complaints? Time? Money? Support? Stability? These are proposed entries for your risk list. Can you counter them with a mitigation statement? Knock off the truly outrageous and unsupported. Keeping them on the list only serves to belittle the critics, and this state of mind will set you back in objectivity. Remember, you don�t have to respect the messenger, just get the message. If there are valid risks to the project, you�ll be better off if you prepare for them. The enemy of your beloved project may be your friend.

Waking up and smelling the coffee � tracking post release issues

We all make mistakes, but do we learn from them? Make sure to set up a process for collecting and addressing the issues that are uncovered after implementation. Any issues that can be addressed, should be. Issues that are clearly beyond the abilities of the software that you have implemented stand as the requirements for your next significant effort. Don�t forget to track them, you�ll need the reminders for the next time you fall in love.

The risk retrospective: a hero's eye view

In a CMM 1 environment where the rewards are doled out according to the pain, risk mitigation plans are eschewed in favor of the risk retrospective as delivered at the release party. As a general rule of thumb the length of the risk list is in direct proportion to the ugliness of the implementation and the lateness of the delivery. A project delivered on time and under budget is dismissed as "easy" while the risks involved in a true disaster are ruminated and repeated down to the minutiae to justify the agony of "success."

This is the list that no one could conjure up during the requirements phase. There were only mumbles and waves available during test planning. But the rumblings about risk grow louder as the weary participants approach and slip past the first drop dead dates.

Think of the risk retrospective as the lipstick for your pig. Keep it in the handbag of your secret project plan.

Why do people do this to themselves? Do we love to suffer? Perhaps. We love a hero, a good story, a tale of challenge and woe. It's not for nothing that the Monty Python troupe got great laughs off of the endless boredom of Chartered Public Accountancy. If they had only been exposed to the daring exploits of Certified Software Test Engineers, or Project Management Professionals; we'd never hear the end of it.

Take note team leaders; it's critical to recognize the accomplishments of your team, lest they create dramas in which they can cast themselves as heroes (or villains)!

But what of the rest of us, those lifetime cube dwellers who may secretly dream of saving it all, against all the odds?

A life, dear ones. We must get ourselves hobbies, take some cube-earned cash and go sky diving, pick ourselves up from in front of the 'net and volunteer somewhere.

Let this be your challenge: change. Make that risk list up front, forgo the drama of the inglorious implementation, just get the thing done.

The project depends on it

Count down to release � is your testing plan ready?

Don�t let implementation catch you by surprise. The testing needs of the release effort are somewhat different than your mid-cycle activities. But don�t panic, either, this is well within your range. First let�s look at where you are in relation to the plan.

Release metrics in the plan � check the obvious.

If you�re lucky you�re doing this research while in the test planning stage. You can be quite optimistic about your readiness for implementation when that stage arrives. If you�re somewhat less lucky you inherited the test plan and it has a bare bones release plan included. You�ll want to take a look at that.

If you�re in a panic it�s probably because your release date is looming and there is no implementation plan. Take a deep breath and read through this. There are no guarantees, but you may find some helpful and sane suggestions here.

The test plan will generally contain the release criteria, and will usually indicate the conditions related to test case status and open defects that must be met prior to release of the code. It may say something like this:

�No critical or high priority defects are open. Any medium or low-priority open defects are signed off as acceptable risks All high priority test cases have passed. User Acceptance Testing has been completed and the user group representative has signed off.�

The standard plan insert is a good start. But no need to stop there. Schedule the meeting to review the project release plan and begin a more thorough review. Schedule first? you ask. Yes, ready or not the review is a must have. Get it on the calendar as soon as you reasonably can.

Defect review � the next step

Remember that the standard is just a start. You need to know where you�re really at. Start with defects. If you�ve been tracking them consistently you�ll have a lot of information there. Start by reviewing the current defect reporting and trendlines and looking for a final spike assoc with regression a.k.a the �bug bounce�

Is the trend otherwise downward? That�s important. In particular your mid and high level defects should be tapering off. If there�s an upward trend in trivial defects this may indicate that your testers have the time to get nit picky � a good sign for release.

Take a moment and analyze defect clustering � make a note of hot spots � we�ll refer to this note later.

Now you�ve got some �known knowns� Here are some activities that you might put on your list:

If there is no bug bounce, check on regression

If the trend is upward, or flat but high, check on over all test progress.

If there are hot spots, check on this area of the test plan for coverage and status

If there are no high priority/critical defects, there was a bug bounce and the trendline is downward, you can feel generally reassured that you�re ready for the release meeting that you scheduled. But don�t stop there, go searching for unknown knowns,

Look for the gaps: review with team members

Communication is paramount, you�re in the final minutes of the final game of the championship series, everyone on the field should be aware of everyone else and aware of the position of the ball at every moment. This is what you need to know:

are all defects accounted for

all statuses correct and up to date

How close are you to the project release metrics?

If you are at the release metrics � it�s time to project the release date, schedule the release activities and make a list of the items that block you from release that might not have been on the release plan.

If you are not at release metrics � d�j� vu! it�s time to project the release date, schedule the release activities and make a list of the items that block you from release that might not have been on the release plan. However in this case you may need to raise the level of awareness on blocking issues.

Set up for awareness.

If your defect tracking system has this feature, set up automatic notice of new defects � possibly by severity or priority.

Face the music on lingering outstanding issues; make explicit plans for the defects that will not be fixed for the release. You may wish to put them in the release notes and/or convert to them to enhancement requests for the next upgrade.

Testing review

Defects aren�t your only metric, look at the test plan and execution metrics. Test reporting can be less straightforward than defect reporting. How you report your testing results can change the best approach for summary reporting.

A tale of two test groups

One test group selected specific test cases from the plan into test sets and tested the same cases over and over again directly linking defects to failed test cases.

They looked for progress over time from consistent test sets and cases.

The second team selected at random from planned functional group test cases using different cases within equivalence classes, looking for progress over time for different test sets and cases.

Different methods, one project, one report, two approaches.

Reporting for the first group could be done at regular intervals reporting the current status of their test set.

Reporting for the second group needed to be done out of the test plan, with explanations for failed test cases that were not run again.

The reporting could be done in a single Excel work book with separate tabs and a central report summary.

What are the important questions when it comes to taking the pulse of your testing status. Here are some suggestions

What is the status of your current testing?

Has the UAT been completed?

Are all the results logged?

When reviewing the planned testing you want to pick out all test cases that have failed and not passed again as well as all test cases not run. When you schedule the final smoke, or exit testing � can you include the stones unturned? Try to include the high priority tests that you just picked out in your review to change the reporting status for them.

In the end you need to certify that any test cases not run cover functionality that has been covered by other tests or are signed off as acceptable risks, required 3rd party certification has been completed and User Acceptance Testing has been completed and the user group representative has signed off.

Final Communication plan

Your communication plan should clearly set out what is going to be communicated to whom at what interval or trigger point. Define a list of finite items for tracking only add to the list critical issues that arise.

Switch from weekly to daily reporting. You may have done this already, but if this is included in your release plan the critical players will know what to expect. This can increase confidence in the upcoming release.

Make sure to target the right people and the right info.

All test team members should be issue aware. Because there are a limited number of critical issues being tracked, this should not be overwhelming to individual team members. It should be possible for any test team member to report on the current status when needed and help them to focus on the important issues for release.

In addition to this all management levels should be statistic aware with details available. The best reporting is high level with drill down information available. Particularly in situations where there will be outstanding issues, or test cases left in a �failed� state, or not run at all, a drill down provides the space for explanations that can address management concerns.

Post implementation test walk through � there�s no second chance.

It is critical that the post implementation testing be thoroughly planned. A walk through of the plan should be on the schedule.

In your release walk through make certain that you spell out and cross check all of the details of the post implementation testing. Everything that you need must be in place when you are releasing new code into a live environment. Be sure to cover:

Materials

Environments

Test cases � review your proposed post release test cases, make sure they�ve already been tested in the test environment.

safe test data � make sure there�s no chance of accidentally using actual real world data. Dummy data in the test system is not necessarily safe in the live environment.

Back out plan � participating in the project level plan

Find out if the baseline code is available to check any concerns

Are you ready?

If you have checked your defects and your test status, put your alert system into place and have rolled out your communication plan, you�re more than half way there. With the release plan review meeting and the post implementation test walk through on the calendar you�ve done a lot to ensure your code release success.

What are you smoking?

Some smart aleck in your organization has suggested �Smoke Testing� you�re wondering just what they have been smoking! Now you�ve been tasked with putting together a �Smoke Test� and explaining to your team just what it is.

You�re in luck. This will be easier than lighting a pipe in a high wind.

First you need to know the purpose of the activity, the rest will follow.

A smoke test is just entrance criteria. It�s the �You must be this tall to ride this ride� of the testing cycle. You�re a test lead or a test manager, or just a tester who is not willing to waste their time on un-testable code. Your purpose is to find the most efficient series of tests to determine the fitness of the code. This means going long and staying high. (Is that where the smoking comes from? No.) Don�t bother with the nit-picky stuff, it has to pass the smoke test first.

That�s �why� now you may be looking for �how.�

There are simple methods for selecting the cases for your Smoke Test suite. First look to the Happy Path. Remember: �If the application can�t walk the Happy Path, it can�t walk at all.�

The next place to look is in the weight or priority applied to your test cases. If your requirements were ranked and the test cases traced to them were ranked you should look at the high level test cases; the test cases linked to the critical requirements. If more than 10% of your test cases are of the highest rank or priority and you need for your smoke test to be streamlined, look for overlaps in functionality, duplications, or equivalents. Can you eliminate down to one case for each area or function? Remember, your smoke test isn�t the last word, it�s the �go� word.

(Streamlined, streams of smoke, is that where we got the word? Not quite.)

Is there a shortcut to finding the showstoppers? Do you have access to insight on the code or on the formatting of messages? Sometimes the fastest way to find the problem is to look into the code itself. If you can shortcut the process by putting some white box testing up front, don�t hesitate to move this to the head of the line. Don�t wait for the testers who only have blind access through the interface to stumble on the problem if you can see it faster by having a developer provide you with access to the internal message formatting. (Getting smoke in your eyes, is that it? No, that's not it either.)

Here are three examples of the usefulness of smoke testing:

You�re a test manager with a team of 7 testers that are already working on various projects. You know that it takes a certain amount of time for them to switch gears and that�s time lost. The code you are getting from the development team on this brand new project isn�t always ready for test. Sometimes you can�t even log in to the new version without it crashing. You�re not willing to �throw it over the wall� to the whole team if you can�t log in and process the most basic transactions. You put together a �Happy Path� through the code to touch the most basic functionality, without visiting the exception cases. If it fails the Happy Path, you can push it back to the development team without breaking the concentration of your testers.

You are a single tester with a detailed 1,000 test case suite ready to go. Each section of the suite plumbs the depths of the functionality of the new development. Running the test cases in order you could take 3 � 4 hours to detect a show stopper bug in a specific module, depending on which section you ran first. But running the 1^st test case from each section will cover the breadth of the development and uncover any show stoppers in the 1^st hour of testing.

Your testing involves white box and black box testing. The black box functional testing will be performed through the user interface by a user group who will be rattled by catastrophic failures. The white box team can see into the code and examine the formatting of the messages that will shape the user group�s experience of the application. If the message formatting isn�t right, the user interface won�t function correctly. It is simple to run a few messages through and examine their formatting to find the errors that are guaranteed to cause failure. This will save you time and energy.

Don�t confuse the purpose of smoke testing with your detailed testing. This is a quick high level exercise that is designed to detect a leak that could turn into a disaster. And by the way, that�s where we got the term: from plumbing!

Efficient and cost effective, smoke testing has become a world-wide standard for finding leaks in sewer and plumbing systems. By forcing smoke-filled air through a sewer or plumbing system, leaks can be quickly detected as smoke escapes through problem areas.

Now you�re ready to smoke!

The Art of Estimation

Main Entry: 1es�ti�mate Pronunciation: es-tə-māt

Function: transitive verb; Etymology: Latin aestimatus, past participle of aestimare to value, estimate

Date: circa 1532

1archaic a: esteem b: appraise 2 a: to judge tentatively or approximately the value, worth, or significance of b: to determine roughly the size, extent, or nature of c: to produce a statement of the approximate cost of

3: judge, conclude

Main Entry: 2es�ti�mate Pronunciation: es-tə-mət

Function: noun

Date: 1552

1: the act of appraising or valuing : calculation 2: an opinion or judgment of the nature, character, or quality of a person or thing <had a high estimate of his abilities> 3 a: a rough or approximate calculation b: a numerical value obtained from a statistical sample and assigned to a population parameter 4: a statement of the cost of work to be done

Main Entry: es�ti�ma�tion

Pronunciation: es-tə-mā-shən

Function: noun

Date: 14th century

It�s about judgment; appraisal; a rough, tentative approximation of value, best handled with calloused hands and a skilled eye. This is not the stuff of laboratories with their over sanitized exactitude. This is about the credibility of well worn boots and measured words. Estimation is an art based on the weight of experience and the credibility of reputation. This is where theoretical science marries the real world. It�s imperfect, it�s challenging, there are wins and losses, but the value gained makes the effort worthwhile.

How do we bring together theory and reality in a way that actually brings value to the project table?

It�s a necessity for planning, and for creating the space necessary to perform proper testing and reduce risk. Yet many testers have little experience in it. Some balk at the suggestion of it. There are those who feel greatly suspicious of it.

Most testing education focuses on uncovering the exact truth, providing statistics that reveal precisely what has happened. Testers exist to validate, not to project. Having invested years of experience in proving the accuracy or inaccuracy of the predictions of others, testers may be rightly loath to engage in such a speculative activity as estimation.

But to gain expertise in this skill will bring to a project valuable planning information that no other team representative can provide and can go a long way in obtaining for testers the one quantity so frequently in short supply: enough time to test. It�s a gem worth panning for.

Let�s examine the known quantities that provide a basis for the exercise.

Perhaps you have no more information than the development timeline and the high level requirements. Is estimation impossible? No, you can still start from there.

Industry Standards

What is the project management standard for estimation? In the absence of all other information 40% of development time plus or minus the level of confidence in the estimate is a good starting point. Where did the magic 40% come from? I have not been able to find a clear scientific reference for this. (I�m open for comments that may point to that source!) However, it is a reliable project management �standard� so you won�t be faulted for using it.

Confidence Level

How sure are you that this is the right amount of time? This level of certainty is your confidence level. It�s not a random number but a number about randomness. Confidence levels are generally used as statistical measures of certainty in results. We can use this to express the randomness of the data we are working with. A confidence level of 50% indicates the estimate is truly random based on pure speculation, with about a 50-50 chance that the time estimated would be the actual outcome. At 75% the odds are still not good - there's a one in four chance that your estimate is meaningless. This describes a situation where only a small portion (less than 25%) of information is available for prediction. Some statisticians consider 90% to be the minimum confidence level for statistically significant results. This would be based on known, repeatable results. If you have historical data that directly relates to your estimate, you can increase your confidence level accordingly.

And in the absence of all other information you cannot really have better than a 50% level of confidence in the estimate. So you start with 40% of dev time, the industry standard, multiply and add half again. Or more if you really aren't sure.

(40% x Development time) = initial estimate

(initial estimate x 50% confidence level) + initial estimate = final estimate

In case you�re wondering, you can save time by combining the two formulas. Just multiply the development time by 1 2/3 or 166.6667.

Increasing Your Confidence: the Work Breakdown Structure

If the test team is in house, the development is standard and there are metrics for set up and test case development, for turnaround, for defect rates and time actually spent running the tests; then you gather all of that information and start revising your estimate.

And multiply by your level of confidence.

As you gather this information you can begin to set up a Work Breakdown Structure (WBS) using Excel. Record every activity that you identify as being a part of the testing process and enter all the data that you have about the time involved. Record the activities even if you lack historical metrics on them.

You now have a list of activities, each of which has an historical metric, or an estimate. Remember to buffer, but be honest about what you are using as your basis. It's better for the process to be transparent, it invites people to volunteer information they may have been keeping to themselves and it instills confidence in your method.

The Value of History
When much of your data is speculative both you and your audience will lack confidence in it. This leaves a lot of room for your test time to be cut back as the project experiences stress. Wherever you can insert real historical data into your equations you will increase the value of your result. If it has never taken less than a week to run a known suite of tests then it is not likely that it will take less than a week in your upcoming test schedule. If the suite must be run, then at least a week must be scheduled. Real gold doesn�t weigh any less in the mountains than it does at the surveyor�s office. Your consistent metrics are your investment for future project planning. If you�ve been putting off gathering them, you can begin now. The project that you are estimating will provide a new historical beginning.

Application in the Real World: Putting the Rubber to the Road
On one of my last assignments I was working with an entirely new test team (theirs!) and some new processes. I had no familiarity with their tests, their environment, or their application. I gave them some training and we immediately began working on a major development project. I needed information from them in order to support project management.

I worked with the new team so that they could help me. I showed them how to realistically measure the amount of work that they covered in a work day, and how to account for time out for meetings and real world interruptions. Testing is never actually accomplished by the minute, and rarely by the hour. In the end I had testers who were pretty savvy about providing information for estimates and I was able to provide estimates to project management: revised at every major milestone, tightened down by actual experience and buffered by confidence level.

It�s not really a science, it�s an art, but you already have more information and experience than you might have suspected. In the end, the more practice you give yourself at this speculation, the better you will become at finding the gold: the right amount of time for the testing you need.

References:

www.m-w.com

http://www.malwarwick.com/learning-resources/confidence-level-calculator.html

But I�m not qualified!

�.you think to yourself while pondering your client�s latest request.

And here is your dilemma: you are being asked to perform some task or project for which you have reason to believe that you are not qualified.

We�ve all been there, well, if you haven�t yet, you will be some day so let�s break this down.

Just how unqualified is unqualified? Is this about stepping up your game? Stepping outside of your comfort zone? Or is this about a completely different professional discipline?

Let�s start with the assumption that you and your consulting firm have correctly represented your experience, certification and educational background. If that�s not the case, then you have a different problem.

If you�ve been asked to step up your game or get out of your comfort zone, then you need to do some math. Given the time and expense it takes to hire a new person, and the outside time expense and effort it would take you to acquire the knowledge and increased skills necessary to complete the task; is this worth it for you and your client?

You�ll do better if you have colleagues who are already masters of the skill set you�re trying to acquire or improve. Having a mentor or at least some �over the shoulder� guidance and advice makes a big difference.

It�s less expensive for you if your consulting firm can help out with some educational assistance. If you�ve been successful with this before then you can have more confidence in taking on the learning task and more likely to get support.

Availability of books and other materials is critical. For example there are plenty of books on SQL, VBScript, working with offshore teams, but companies like Mercury who provide important software programs (Quality Center, LoadRunnner etc.) keep tight control on product information and training. Are you trying to learn something that cannot be cheaply or self taught?

Perhaps you are dealing with a more serious situation. Your client is impressed with your breadth of knowledge in Software Testing and has an idea that you could perform a task that is well outside of your discipline. You simply are not qualified to do what they want and they don�t want to hire another party. They are determined to accomplish this and want your professional help.

If you really do have a breadth of experience then you may be qualified to help them more clearly understand their needs and maybe you can help them prepare themselves for seeking the additional help that matches those needs.

Most testers are dogged if not expert researchers and this can be an invaluable resource for a client who is really lost in the woods. You can be the person who pulls together the requirements, outlines the standards that are a part of that other professional discipline and proffers the alternatives that give your client a real educated choice.

Don�t be shy about letting your consulting company know about your client�s new needs either. An good account manager is trained to follow up on such a lead and provide your client with convincing and (hopefully) reputable arguments for getting the job done �right.� And your company is generally invested in your credibility. They won�t want you to take on a task at which you would inevitably fail.

Staying honest about your limitations, while still being willing to go the extra mile to make sure that your client gets their needs met in the best possible way won�t hurt your opportunities. In fact, it�s more likely to enhance them.

Finding your way to the Happy Path

Maybe you don�t need to find your way to the Happy Path. Maybe you�re already on the Happy Path. Perhaps when you suggested to your client user group that they define the Happy Path they replied:

Oh yes, that�s outlined in the first Visio diagram on the project SharePoint site. Well, actually, that�s just a link to the actual diagram on our Standards site, maintained by our Standards Manager, Jennifer Wong-Sharma.

�at which moment you found yourself looking into the calm and intelligent eyes of Ms. Wong-Sharma who seemed to understand everything that you needed to get on your way with your test plan for your software development project.
Then you would find yourself on the Happy Path, indeed.
But perhaps your experience has been less optimal.
Perhaps when you mentioned this �Happy Path� you got this:

But all paths are the happy path!

Don�t we have to test everything?

What is a path?

This is perhaps more typical. In this scenario I find myself sitting back and asking myself just why it is that I need a Happy Path and how I communicate this value to my clients.

As allegedly defined by Richard Harrah of HP the Happy Path is:

a default scenario with no exceptional conditions

Because of the lack of exceptions, conflicts, challenges, the path, (or the data flowing over it), is happy.

It is the �default scenario� because it is the thing that the software is being designed to do. If it is a purchasing software, this is a purchasing scenario. If it is a word processing software, this is a document creation scenario. If the software project has a title, or if the end result were put in a package and given a name, this would give you a clue as to the default scenario.

In some cases it is the money path. If this is an application related to your client�s business, then the Happy Path is where the client is making the most money.

In other cases it follows the 80/20 rule � it is the path that handles 80% (or more) of the transactions that flow through this system.

In any case, it provides the project with an organizing principle. Everything exists to serve the primary function � the Happy Path.

What it is not, is a path that shows you other important functions, including error handling, even though these must also be tested. These functions exist to serve the primary function.
When you have this defined, you have the basis for prioritizing your test cases, and, as long as you have the test cases traced back to your functions and workflows, you have a quick list of test cases that provide you with a smoke test, or entrance/exit criteria. If the application can�t walk the Happy Path, it can�t walk at all.

When you define your Happy Path you provide the cornerstone for organizing many of your testing activities, and for a project that needs organization, that�s a happy thing indeed.