Should testers execute their own test cases?

Ask this question and you will get a barrage of differing views all with their reasons for a “yes” or “no” answer.  It seems that this is an area that testers and test managers just can’t agree on.  

So maybe there is something we can agree on, or at least debate, without having the poles apart yes/no extreme to the question.

Can we agree that test cases should be created?  No, we can’t. James Bach would argue that test cases have no place in doing testing.  If I read him correctly, he believes that formally written test cases have limited value and can actually be counter-productive to good testing and their only use is if there are legal reasons to create them (apologies, James, if I have mis-represented your view and you ever read this).  I have a lot of sympathy with that view and like to reduce (note, not eliminate) the number of test cases that my team writes to a minimum level necessary.  Key phrase there is “minimum level necessary”.  I believe there are occasions when the most effective way of testing something is by writing a test case.

Ok, so we can’t agree on whether we should write test cases, hmm, where to go from here?

Perhaps we can agree that tests need to be designed?  If we take the definition of “designed” as meaning that the tests are thought about before actually executing, then I think we can agree.  Even if you are doing Exploratory testing you are designing tests, by jottings on pads of paper, frameworks in your head or bright ideas that just pop-up as you learn more about the system you are testing.  Ad hoc tests are not designed, but I would argue that Ad hoc testing isn’t testing, it is playing around and has no place in the professional tester’s thinking.

So, if we can agree that tests need to be designed, then what is the implementation of that design?  

Before we determine that, maybe we need to look at what is the purpose of the test.  The widest view of the purpose of any test is to find a bug.  OK, we can argue about risk, control, confidence level, etc. but at the end of the day we are trying to find bugs.  But, it isn’t quite as simple as that.  From a project perspective, yes you are trying to find bugs, but there are supplementary purposes which are valuable to the test team, the project and the company.  Some examples:

·        Teaching a tester how to test.  The only way I know of teaching someone how to test is to get them to test.  Training courses are all very well, but they only give pointers on how to test.

·        Teaching a tester about the product.  It would be nice if every tester knew everything about a product before they started to test it.  Some testers may, but some (maybe all) will not.

·        Improving a tester’s skill set.  This may be in terms of getting the tester to use a previously unknown/unused technique; it may be in terms of getting them to lead a test effort.

The implementation of the test design, therefore, should be dependent on the purposes of the test. 

But we still do not have a full answer on how to implement the test design, because now we should consider what is the approach taken to testing?  By this I mean are you a test case shop, an Exploratory shop, unstructured or a mixture of approaches.  Your implementation of the test design will depend on your approach and your approach may change depending on the product you are testing this week.

When you have implemented your test design, then you need to execute the implementation.  Part (a small part) of me argues that if your design is good, then your method of execution is irrelevant, because you have done all the hard work of thinking and the rest is mechanical.  The rest (and more dominant part) of me argues that all you have done is laid a framework, no matter how good your test design is, the system under test is going to surprise you and you will not have covered everything you find you need to cover.

If that is true, that your test design implementation is only a framework, how do you determine who executes the test?  That decision should be made on who is the most appropriate person to execute the test, based, again, on what is the purpose of the test and the method of test design implementation.

No doubt in twenty years time people will still be arguing about whether the test case author should also execute the test.  Me?  I’ll stick with saying that “it depends” and take each situation on its merits.

I like IDMTDT testing, it can be a rich source of errors and can show up data integrity faults and usability problems that you might not otherwise find.  The big advantage is that you can then see how the system handles CRUD activities in unusual combinations.

Oh, sorry, IDMTDT stands for “I didn’t mean to do that” and is a set of tests that try to recover from when the user has done something stupid and wants to undo somewhere down the line in the Business process. 

A simple example:

User creates a new record for a new customer

User enters name/address/Post code

User then creates new accounts record for new customer

User puts credit limit on new customer account

User enters sales representatives name

User activates new customer.

At this point, User finds that the Post code is mistyped.  How do you recover?  If the system allows an edit of the customer field, then it is easy, but lets say, for the sake of the example, that there is no edit function for the Post code, it maybe a primary key in the database that is uneditable.  Let us also assume that deletion of customer records is not allowed as the system requirement is to keep all records for audit purposes and the design assumes that the user does not make this type of mistake. 

The recovery from this type of problem in a real world situation can be a nightmare (I have seen this type of scenario in a number of systems).  The testing can, however, be quite fun as you have a number of points at which you can try to recover the situation; after the new record created, after the account set up, after the sales rep entered, after activation.  Each of these scenarios will have a different set of tests and may involve exercising different functionality.

Planning and scripting such testing tends to be difficult as it is often the type of scenario not covered well in requirements or business scenarios (because people don’t make mistakes, do they?), so the best technique I have found is to include it in Exploratory tests.

It is a type of testing that can be forgotten or is done haphazardly.  It also, as a general guideline, should be done once the software is relatively stable as you are not trying to find functionality bugs, but business flow bugs.

Following my entry entitled “Are you positive about being negative”, Joe Strazzere asked what do I call the tests that we execute when we run out of time.  This is a good question and one that I should have answered in the original entry as Joe has correctly pointed out that I now don’t have a name for them.  

One of the reasons for not putting a tag on this type of testing is that it can cloud the thinking when determining what tests to run.  “Happy path”, for example, would indicate a certain type of test.

If you need some type of “tag”, the best I can come up with is MAT, which stands for “Most Appropriate Tests”.  Let me explain what I mean.

Whenever you are short of time for some tests you have to determine what tests you will run to make the best use of the time available. The thought processes go along the lines of:

• What are the risk areas of the system
• What are the areas that have spawned the most bugs recently
• What are the most business critical areas
• What are the areas that would be most embarrassing if they failed
• What has changed recently and therefore could have de-stabilised the system

So, the upshot is that I can’t give these tests a generic term, the answer is “it depends”, you should run your MAT suite!

Are you positive about being negative?

I don’t believe there is a distinction between Positive testing and Negative testing.  I don’t believe that there is such a test as a “Positive” or “Negative” test.

There, I have said it and the heavens haven’t opened up and I haven’t been hit by a thunderbolt.

You often hear the question “What is negative testing?” and you get various answers.

Some definitions I have read/heard:

1. Those test that are designed to return an error, using invalid input
2. Tests that are designed to cause the system to crash. 
3. Tests that are designed to make the system do something “off spec"

With reference to definition 1, so if I have an integer field and I put in an alpha character I expect to get an error, yes?  So what is negative about that test?  I have made a positive input and expect a positive response, i.e. an error message. 

With reference to definition 2, so, if I set up corrupt data I expect the system to crash, yes?  Again what is negative about that?  I have made a positive decision to try to force the system into an error condition to see how it handles it.  Where is the negativity?

With reference to definition 3, so, I am doing something to the system that might make it do something unusual.  Is this negative?  I have designed tests that try to force the system to do something it shouldn’t.  This is the closest that I can come to saying it is a “negative” test, but even here I am making a positive decision.

If one of the primary reasons we test is to find bugs, then we should be designing tests that are targeted at finding bugs.  Any means at the tester’s disposal should be used to find those bugs, use whatever techniques you have in your toolbox, whatever tools you have and you will execute “just tests”, not positive, not negative.

What does it matter?  My concern with classifying tests as positive or negative is that it is an artificial classification that clouds tester thinking.  There are no positive or negative tests, only tests that exercise the system looking for bugs.  I don’t understand the reason why testers want to distinguish between positive and negative, what benefit does it give?

I have heard people say “we are running out of time, only run your positive tests”.  But which are those?  I have a set of tests, I can’t differentiate them according to positive or negative.  What people usually mean by that statement is “Only run those tests that exercise the system the way the user will normally use it”.  OK, I can identify those tests, they are sometimes called “Happy path” tests, and we always create those, but they are a small percentage of the tests we prepare, I estimate less than 10% of tests fall into that category.  Are they positive tests?  By some definitions, yes, but what if one (or more) fails? Does that make it a negative test?  Again, does it matter?  A failure is a failure whatever classification is used for the test.

I don’t want a tester to go off and run a set of “positive” tests, I want him/her to execute tests that exercise the system in such a way that have a reasonable degree of probability of finding bugs.  That should be by exercising the system in the way the user will normally do it and doing other things as well.

I guess that I am not going to suddenly stop all the testers in the world from using “Positive” and “Negative” tags for tests, all I can do is try to stop my own testers!

Well, philk10 has been nagging me to write up last week’s SIGIST, so here goes.

Just for those who don’t know what this is, it is the British Computer Society Special Interest Group In Software Testing (heck of a mouthful, so SIGIST will do).  A one day conference is held every quarter and the quality of the speakers has improved considerably over the last few years, including some world known people.  Last week’s was well attended, especially considering there was a very important football match on that day.

The SIGIST this time had Johanna Rothman as the guest speaker.  She did three sessions, so she must have been really tired by the end of the day!

First session was entitled “Are you Hiring Yesterday’s Testers?”.  The presentation focussed on what a first class tester is and how to recognise the difference between first class and second class testers.  This is not concerned so much with the ability and skill of the individual tester, but attitude, maturity and skill of the hirer.  The following list, taken directly from the conference handouts, shows what first class testers looks like:

• Has a wide variety of testing skills
• Finds the most atrocious problems before release
• Developers enjoy working with the testers
• Learn about risks and other product information
• Can quantify the risk of a release
• Accurately predict necessary testing time and can explain why
• Understand product release is a business decision
• Supply enough information to predict rework and help projects complete on time

The following list, again taken from the handout, shows what second class testers look like:

• Routinely excluded from requirements or design meetings
• Resort to eavesdropping to hear information about the product
• Request for tools postponed or ignored
• Training budget significantly less that developer’s budget
• Interchangeable, i.e. they have such similar skills it doesn’t matter who you assign to which products
• Only work with developers on code artifacts because they either aren’t brought into the project early or they don’t know enough about the early phases to supply feedback
• Work all hours because the developers have so many defects to fix

Do you recognise any of those attributes from either list in your group?

Johanna went on to discuss how to create a first class group, which boiled down to making sure that the first class list above was achieved.

She also talked a bit about how to hire first class testers and it was a very short précis of her book “Hiring the Best Knowledge Workers, Techies and Nerds:  The Secrets and Science of Hiring Technical people”.

It was an entertaining presentation that I saw a number of people nodding their heads in agreement (no, they weren’t falling asleep!)

Johanna’s second session was a workshop entitled “How to Become a Sought-After Tester or Test Manager”.  This was a workshop designed to make you evaluate your skills and determine an action plan to improve them.  I think it would have worked well with a small group, but the group was large and it didn’t have the dynamics that a small workshop would have had.  Interesting, but not as useful as I had expected.

Johanna had the last session of the day entitled “Successful Software Management: 15 Lessons Learned”.  This was a superb presentation and I can’t do it justice here.  Basically, Johanna was going through what makes you successful as a Technical Manager.  The 15 lessons were:

1. Know what they pay you to do
2. Plan the work
3. Accept only on Number 1 priority at a time
4. Commit to projects after asking your staff
5. Hire the best people for the job
6. Preserve good teams
7. Avoid micromanaging or inflicting help
8. Treat people individually and with respect
9. Meet weekly with each person
10. Plan training time in the workweek
11. Fire people who can’t do the work
12. Emphasise results, not time
13. Admit your mistakes
14. Recognise and reward good work
15. Manage yourself

The presentation was peppered with her own experiences, mistakes and successes and whilst most managers know all these things, how many of us actually do ALL of them?

I went to two other presentations, the first was by Isabel Evans, entitled “A Balanced Scorecard Approach for Assessing Test Value and Success”.  This was interesting as I had heard of and seen the Balanced Scorecard Approach, but what Isabel has done is to adapt the approach for testing and how we can apply our metrics to show our value and success to the organisation.  I could see how it could be useful and I could also see that some of it might take a lot of time to put together, so I still have to think through how I can use what Isabel said in my own shop.  Her message is that what we report must be of use to the reader in their own language.  An example:  We are good at producing defect statistics, X raised this week, Y still open, Z closed, but what does that mean?  We should be couching our report in language that the reader is interested in, e.g. “the defect rate is such that the release will not be stable by next week”, or “the cost of rework due to the defect level is £x.  Her basic conclusions were as follows:

• Any organisation needs a balance of financial, customer, internal and innovation measures
• Test Teams exist to provide information
• Test Teams’ customers include the organisation, the project, the managers, the builders, the supporters, the users and customers and other measurement groups
• Our measurement and reporting help us
• To help our customers our measurement and reports need to reflect their goals, targets, indicators and concerns

The second presentation was by Graham Thomas, entitled “Seven key measures for Software Testing”.  This presentation took us through a horrendous example of a weekly test report that he had found, he showed the problems with it and then presented seven measures that help the weekly report become useful.  To be honest, there was nothing in the presentation I didn’t know and don’t use in one form or another, but Graham was a good speaker and made it entertaining.  His main focus was on showing test progress, script progress, risk profile, risk mitigation, Fault S-curve, Environment availability and Coverage.

The were other sessions I didn’t go to, because they clashed with the ones I did go to.

Conclusion: A good day, I learnt a lot.  This SIGIST was very test management biased whereas other SIGISTs have been very testing biased, so it made a nice change.  Oh, and England did win the football match…

A sledge hammer to crack a nut?

We are about to start testing a major enhancement to the system.  Development will be delivering this enhancement in two drops, firstly the back end and a month later the client end.  This has meant that we have had to write a utility to create the packets that the client would have sent to the host in order to test it.  This Perl utility gets responses from the tester, formats the responses and saves them away to a file for processing on the host.

My “plan” was to then use QuickTest Pro to be able to create many of these files and files with many packets, using a spreadsheet to drive the QTP script.  I estimated that it would take about a week to get this working as I wanted it.  Unfortunately all my QTP resources were busy on other things and could not get to it.  Panic was starting to set in until I was in the shower this morning and had a “Eureka” moment – or more likely I realised I was being stupid!  I could do the self same thing with a small VB script in Excel.  When I got into work, I spent hour and a half putting the script together, testing it and hey presto everything I needed.

I know a lot of people state home grown tools are often the best, but I wonder how often we plan to use the tool we have without really thinking about the best way of doing it?  I saved my team a week’s work and got something for free – well, I am a Manager, so my time doesn’t count J.

Selling Exploratory Testing

One of the difficulties I expected to have when introducing Exploratory Testing was having to sell Exploratory Testing to the customer and to my own management.  What I didn’t expect was having to sell it to my own testers.  Some of them had used ET before, some had not.  Those that had needed no persuading, they already knew the benefits that ET gave.  Some of those that had not used it before were very sceptical and could not see how you could test something without scripting it first.  

This surprised me, as testers, by their very nature, explore the system under test whether they are scripting or not (see James Bach’s articles on this on www.satisfice.com).  I tried to determine why there was this resistance to a way of testing.  The answers I came up with are, inevitably, based on a very small sample of testers (c40) and are, therefore, just my view on what I saw.  [Perhaps someone will want to pay me for a larger study J]

1) The way a lot of us have been trained in IT has been to document first, then do.  Just “doing” is considered to be unprofessional and cavalier, akin to hacking.  So just testing can have an uncomfortable feel to someone trained that way.  

2) There is the attitude that if you can’t see what I have done, then you can’t measure how good I am.  This attitude stems from the CYA attitude that comes from bitter experience that if something goes wrong, you had better have some evidence to show that you did your job.

3) In my own shop, we had the customer crawling all over us in the early days, distrust was rife, so everything had to be specified minutely and thoroughly reviewed by the customer.  It was drummed into the test team that documentation was everything.  This legacy meant that changing the way of working was difficult purely because it was a change and some people don’t like change.

4) Related to point 3, there is a fear of the unknown.  Testers who had never done ET before were nervous of failing when doing something new.  Even though they were experienced testers, it was an approach that they had to learn.

How did I get round this resistance?  

Myself and one of my Test Team Leaders who was experience in ET did a number of courses to the teams on what ET is, what the benefits are and how we were going to implement it on our project.  As part of the course, the team had to do some ET.  This ET was on a piece of the system that everyone knew, as it is core to the system and supposedly very stable.  Each of the courses found problems in this part of the system that we didn’t know were there.  None of the problems were serious, none would have caused us to stop shipping, but it the made the point that ET can find problems that had not been found.

We also put in place Session Based Testing (as defined by Jonathan Bach) and this helped, because the testers could see that there is documentation associated with ET and that the documentation was useful, even if it is totally different than what they were used to.

Now ET is a major weapon in our armoury and you wonder how we ever got along without doing it in the past.

Implementation of Test Ideas.

INTRODUCTION

At the British Computer Society Special Interest Group in Software Testing last year, I heard Robert Sabourin (www.amibug.com)  talk about Test Ideas and how to workshop them.  I am sure many test teams use a brainstorming approach to determining what to test, but Robert has a structured approach to brainstorming which looked interesting and I decided I would give it a go in my own team.  I have now been running these workshops for some months so I thought I would give some feedback and recommendations from our experience of using them.

Test Ideas workshops are a form of brainstorming that helps to structure the thinking of the testers and allows fairly easy documentation of the test ideas.  The Following section is my précis and downright plagiarism of Robert’s article “What Not to test” a link to which is included below (although I just tried the link and it didn’t work).  This document is what I used to provide guidelines to my test team.

TEST IDEAS

A test idea is a description of a kind of test that needs to be run. These are documented in a table as follows:

The fields in the tables are:

ID is just a reference number.