Peter Nairn

Are you positive about being negative?

Posted on Thu 1 Feb 2007 at 12:44 in Testing

 

Are you positive about being negative?

 

I don’t believe there is a distinction between Positive testing and Negative testing.  I don’t believe that there is such a test as a “Positive” or “Negative” test.

 

There, I have said it and the heavens haven’t opened up and I haven’t been hit by a thunderbolt.

 

You often hear the question “What is negative testing?” and you get various answers.

 

Some definitions I have read/heard:

 

  1. Those test that are designed to return an error, using invalid input
  2. Tests that are designed to cause the system to crash. 
  3. Tests that are designed to make the system do something “off spec"

 

With reference to definition 1, so if I have an integer field and I put in an alpha character I expect to get an error, yes?  So what is negative about that test?  I have made a positive input and expect a positive response, i.e. an error message. 

 

With reference to definition 2, so, if I set up corrupt data I expect the system to crash, yes?  Again what is negative about that?  I have made a positive decision to try to force the system into an error condition to see how it handles it.  Where is the negativity?

 

With reference to definition 3, so, I am doing something to the system that might make it do something unusual.  Is this negative?  I have designed tests that try to force the system to do something it shouldn’t.  This is the closest that I can come to saying it is a “negative” test, but even here I am making a positive decision.

 

If one of the primary reasons we test is to find bugs, then we should be designing tests that are targeted at finding bugs.  Any means at the tester’s disposal should be used to find those bugs, use whatever techniques you have in your toolbox, whatever tools you have and you will execute “just tests”, not positive, not negative.

 

What does it matter?  My concern with classifying tests as positive or negative is that it is an artificial classification that clouds tester thinking.  There are no positive or negative tests, only tests that exercise the system looking for bugs.  I don’t understand the reason why testers want to distinguish between positive and negative, what benefit does it give?

 

I have heard people say “we are running out of time, only run your positive tests”.  But which are those?  I have a set of tests, I can’t differentiate them according to positive or negative.  What people usually mean by that statement is “Only run those tests that exercise the system the way the user will normally use it”.  OK, I can identify those tests, they are sometimes called “Happy path” tests, and we always create those, but they are a small percentage of the tests we prepare, I estimate less than 10% of tests fall into that category.  Are they positive tests?  By some definitions, yes, but what if one (or more) fails? Does that make it a negative test?  Again, does it matter?  A failure is a failure whatever classification is used for the test.

 

I don’t want a tester to go off and run a set of “positive” tests, I want him/her to execute tests that exercise the system in such a way that have a reasonable degree of probability of finding bugs.  That should be by exercising the system in the way the user will normally do it and doing other things as well.

 

I guess that I am not going to suddenly stop all the testers in the world from using “Positive” and “Negative” tags for tests, all I can do is try to stop my own testers!

 

You nailed it!

Posted on Thu 1 Feb 2007 at 03:01 by JakeBrake
Amen! Amen!
I am guessing that this originated from some schedule-pinched manager who was pinched by the upper echelons whom were pinched by the ever-present over-promise dude.
I agree that it tends to cloud judgement. Imagine the time wasted by this layer of thinking. I would wonder how many defects went out the door because management forced a positive/negative attitude and then insisted on "happy path" tests only, thus allowing many defects to escape out into the end-user world.

Happy Path versus Positive Tests?

Posted on Thu 1 Feb 2007 at 06:35 by strazzerj
So rather than Positive / Negative,
you'd prefer the term Happy Path / Sad Path,
or Sunny Day / Rainy Day?

Joe & Sunny Day / Rainy Day

Posted on Thu 1 Feb 2007 at 08:21 by JakeBrake
Joe might be onto something.
One year hence:
"Let Joe Guide You through Rainy Day Testing"
ISBN - Reserved

Two years hence at SQAF...
Wut iz diff btwn Sny day n Rny Day testing?

test drive

Posted on Thu 1 Feb 2007 at 11:32 by philk10
For those people that just want happy/positive/sunny tests run, ask them what sort of test drive they do when buying a car - do they drive it down a straight road for a mile or would they try roads with curves, reversing, an emergency stop...

NOooooooo...

Posted on Fri 2 Feb 2007 at 01:24 by PeteNairn
Joe, I don't want to call tests Happy paths, sad tests or anything like that - that is what I want to get away from!!

Perhaps we should start up a banned terminology group?

OK

Posted on Fri 2 Feb 2007 at 06:05 by strazzerj
Ok, so your group no longer categorizes tests into two groups (positive/negative, happy/sad, sunny/rainy, whatever).

Now, when folks come to you and say, in essence - "We are running short of time. Please run some meaningful subset of your tests, instead of all of them." will you just say "No"?

Or will you have some other way to divide them into "the ones I really feel are necessary" versus "the ones I can live without".

And if so, what will you call *those* two categories?

Timely

Posted on Fri 9 Feb 2007 at 03:39 by dsynadinos
Ha! Before I read this blog entry, I discussed this very topic with a few of our testers earlier this week. I explained that I didn't see the distinction between "positive" and "negative" testing. When testing, you really only have "expected" and "unexpected" results.

And, in response to strazzerj:
Why must tests be categorized into "positive/negative", or "happy/sad", or "sunny/rainy"? Why only 2 categories?

Tests can be placed into many different categories based on many different criteria. What is the priority of the test (maybe calculated by impact + probability, or some other agreed-upon definition)? How long does it take to execute the test? What tests exercise new vs. old functionality?

The question(s) you decide to answer will likely results in different subsets of tests.

Edited by dsynadinos on Fri 9 Feb 2007 at 07:45

Last Page | Page 18 of 41 | Next Page

RSS feed

- Subscribe