Rainbow Testing

Regression testing – the Ugly Betty of Testing

I was once asked by a high level manager whether failing a regression test meant that there was regression, or there was not. Clearly we have failed to communicate.

Of course, we are checking to ensure that there has not been regression.

No regression, no backsliding, retrogression, relapse nor degeneration! We move forward without losing any ground.

Or so it is proposed.

Regression testing is the cousin to automation, the dowdy step sister of all the ***ier techniques applied to new code (which is the rich vein of imperfection that temps avid testers). Not so for regression testing. Few of the ambitious long for regression testing.

It is the stalwart helpmate of version control.

Regression testing, in an optimal environment, will uncover few defects, few mysteries. Its foremost weapon is so dull it is called a Smoke Test. It is so predictable that its secondary use is as a training tool for new testers. It presupposes a thorough and unshakeable knowledge of the base of the system under test.

It is boring.

Besides this it still requires constant maintenance. Little surprise then that it can be difficult to charm your best testers into taking it on. Testers generally lust after breakage the way the Cookie Monster craves chocolate chips.  Regression testing at it’s best provides little of that type of excitement.

Then what is exciting about Regression Testing?

Regression Testing is a true thrill to:

·      anyone who is stuck in a CMM level 1 environment and knows in their heart that it doesn’t have to be like that;

·      any tester who had the experience of dropping an hour’s worth of work into fresh code only to find out that the whole thing is invalid because the code drop had a fatal flaw;

·       anyone who has had to spend hours refreshing test data that got burned off against a bad version

·      any cross over automation testing geek who gets a charge out of setting the stage for lighting fast diagnostics

You get the picture. Regression Testing appeals to the folks who are drawn to testing because they like straightening things up and sorting things out.

Some testers like seeing the defect count. Others like taming the Wild West of the Software Development Lifecycle.

If you decided to take on the Regression Testing then Ugly Betty just bought herself a saloon.

Your six guns are going to be those gun metal grey smoke tests. Oil them up. You are looking for fast, brutal paths through the system. No depth here, just run that bullet through every ten gallon hat you have lined up easy, without any fuss.

If this is an order system, you want to have a test that runs an order, end to end, using the most typical choices possible. Fall in love with default processing for this one.  You want an answer to this question in 3 notes or less: Is the system testable?

If you cannot get the simplest order through the system, the simplest user path completed end to end, stop the piano player and throw that buckaroo out of the bar. Stopping a bad build with a good smoke test is as easy as licking butter off of a knife. Congratulations, Regression Tester! You just saved your team and your company hundreds of dollars in wasted time.

What’s next?

If the system passes the smoke tests you can, if that’s your process, hand it over to the eager testers that forge their way across new lands in search of problems and unmet requirements, but you’ve still got work to do, and the faster the better.

Remember that the goal of testing is to find the most significant problems soonest. Leave it to the rest of the team to be vetting the highest level requirements etc for the new development for you this means something different, and possibly something undefined.  

What’s the most significant problem?

Your research for your Regression Suite should have included a chat with some of the good people in Finance. What’s the company’s money maker? It needs to be tested. Find someone who can run you reports on transaction distribution. What is the user doing the most? The point of Regression Testing is to validate that what has been happening can still happen. Put the things that happen the soonest and the most often at the top of the list.

It’s inevitable in software development; the user will be disappointed somehow, some day. It’s best if that’s not their first experience, or even in their first 5 minutes on the new code.  

Tight Scrouging

Here’s where you get to hoe the corn. This is the real test of what you know, or what you can find out, about the end to end system. A first class regression test is going to have a case set up to hit nearly every single system end to end. Again, no depth, but in this case, or these cases (you may need more than one for coverage) the object is breadth.  

Most systems have certain exception criteria that will send the user path off into a specific function or subsystem based on certain specific criteria. The real ace for this set of cases is an input that will send the user through as many of those as possible.

Use a spreadsheet, use a mentor, work with a team, but make sure you have a set of tests that hit all of the functions and subsystems with the minimum amount of input and individual cases.

Remember those scenes in your favorite old westerns where the sharp shooter looks in a mirror, aims over their shoulder and bounces a bullet off of every hard surface in the saloon before blowing the hat off of the skeptic? You are that sharp shooter. Go ahead and blow their hats off.

Recording and reporting

You are going to have to change your perfect test. The system will change and grow and you are going to have to move with it. Make sure that you have classified your test anatomy properly. Be clear about your inputs, your test data. If it’s easy to find it will be easy to update… and easy to automate. In fact, take some time and talk to the automation tester that you are working with. You’ll both save a lot of time and money by having overlapping approaches.

Good coders leave clear extensive comments that explain what they were intending when they wrote the code. Don’t you be shy about leaving clues for the history books. Make sure that your test format allows for commenting. If the case relates to a specific requirement or old defect, record that.

Make sure that you can sort out the test cases that target specific functions, inputs or subsystems. This allows you to run subsets of the larger test suite or make quick updates across the Regression Test when a specific area changes.

How does the Configuration Management (CM) system classify the parts and pieces of the system? Does your system dovetail with that?

What metrics need to be collected from testing? Does your Regression Test provide input for that?

These are the kind of questions that can help you double check your Regression Test strategy against other systems in use in the enterprise.

In the end, while the denizens of your saloon have their own high adventure, they probably won’t give a thought to the work that goes into keeping the environment open for that fun. That is as it should be.

Let the adventure be the new code, not the same old chaos. You’ve now got a tight grip on a bucking bronco and that’s something to be proud of in this Wild West. Forget Ugly Betty, you are the Money Making Mama of this here town.