Implementation of Test Ideas.

INTRODUCTION

At the British Computer Society Special Interest Group in Software Testing last year, I heard Robert Sabourin (www.amibug.com)  talk about Test Ideas and how to workshop them.  I am sure many test teams use a brainstorming approach to determining what to test, but Robert has a structured approach to brainstorming which looked interesting and I decided I would give it a go in my own team.  I have now been running these workshops for some months so I thought I would give some feedback and recommendations from our experience of using them.

Test Ideas workshops are a form of brainstorming that helps to structure the thinking of the testers and allows fairly easy documentation of the test ideas.  The Following section is my précis and downright plagiarism of Robert’s article “What Not to test” a link to which is included below (although I just tried the link and it didn’t work).  This document is what I used to provide guidelines to my test team.

TEST IDEAS

A test idea is a description of a kind of test that needs to be run. These are documented in a table as follows:

The fields in the tables are:

ID is just a reference number.

Source is the source of the test idea. There are many sources for these testing ideas; some are shown here:

Description briefly details each idea. Notice that the ideas aren’t specific enough to be test procedures, but they are specific enough that a tester easily could flesh them out. We will invest in detailing an idea more specifically if (and only if) we decide to use it. If we defer or reject a testing idea, there is very little benefit to further detailing or elaboration. Imagine the waste of spending several days detailing a test procedure only to decide not to implement the associated testing idea. In testing, size matters—hence the field size.

Size. Consider the size of a testing idea to be an estimate of the reasonable amount of effort you would have to invest to satisfactorily implement that idea given the state of the software under test and the skills and competency of the testing team. Small is a testing idea that takes less than ninety minutes to implement. Medium is a testing idea that takes a day to implement, and large is a testing idea that takes about a week to implement. If test ideas are larger than large, we split them or otherwise reorganize them.

Importance. The bottom line is that a test idea is not important if the bug it finds will not be fixed. For example, if we decide not to fix spelling mistakes in final system testing, then any test idea related to spell checking is unimportant. A testing idea is important when the information it provides contributes to the decision to deploy or ship the software. High implies knowledge of the test passing was critical and low implies a decision to ship or deploy the software could be made without knowledge of the test result. Anything in between is considered medium.

CAPTURE TESTING IDEAS

To fill in the table, a structured brainstorming session is held  which is designed to capture testing ideas. We can decide not to implement a testing idea based on the information we have now, but if new information comes our way in the future, we can reconsider using the testing idea. There are no wrong testing ideas. They all have some value even if they are not implemented.

Test Idea Triage

Test ideas are not static. As testing progresses, new ideas are discovered. As testers and developers start using the application from an end-user perspective, usage, scenario, and inter-operation testing ideas are generated.

Further Information

See http://www.ece.mcgill.ca/~info429/2004_10_07_What_not_to_Test.pdf

My team’s experience

Test Ideas Workshop

Everyone in the workshop is given index cards and is given 20 minutes to write down at least 4 ideas against each type of test.

Once complete, the cards are collected up and posted on flip charts and the moderator reads them out, asks for clarification if the idea is unclear, but no discussion is allowed at this point.

Once read out, the floor is open for further ideas.  These ideas are written up on the flip charts as they occur.  Discussion is allowed at this point, not on the merits of the ideas but on further possibilities and ideas.  This sparks off further ideas which are also captured.

When the workshop has run out of ideas, the moderator runs though each idea and the workshop agrees Importance and size and eliminates the duplicates.  Typically, this also generates some new ideas and these are also captured.

Finally, four “volunteers” are assigned to write up each of the sets of flip charts in the standard format.

Our Method

We took the Robert’s method and decided to apply it as Robert stated, with two exceptions:

·         Robert says that you write one idea on one index card (he has pre-printed cards).  We decided this was too messy when you had a number of people in the workshop, so we allowed more than one idea on one card.

·         Robert suggests that the Size and Importance is determined by the person writing the idea, we decided that we would not do that, we would determine the size and importance as a group at the end so that we were not arguing about size or importance when we should be coming up with ideas.

What was good?

It was abundantly clear from the very first workshop that this method beat brainstorming hands down.  The number of ideas was greater and the quality of those ideas was superior. 

The team in the workshop were enthused about the project and got their focus on the testing much earlier than it would have been otherwise.

You have a very good starting point for doing your test preparation

What do you need to beware of?

As with all types of workshops or brainstorming sessions, you need to be aware of the dominant personality problem.  This is where the 20 minute session at the beginning helps a great deal because even the quieter members of the team have a say right at the start, getting them involved.

You need a good moderator who can control the workshop.  Everyone talking at once, arguments, sub groups forming are all bad for the overall workshop and the moderator needs to be strong enough to prevent the bad things happening.

Time flies!  We scheduled 1.5 hours for a workshop that took over 3 hours.  It will always take you longer than you think.

Make sure that the ideas are written up as soon as possible after the workshop so that everyone can see them.  This keeps the momentum going and allows people to critique the write-up.

Conclusions

These workshops work well.  They give the testing a great kick start.  

You can use the test ideas in a number of ways, you can produce manual scripts from them, automated scripts from them, you can use each idea or a group of ideas as session objectives for Exploratory Testing.

I would highly recommend this method and, if Robert Sabourin ever reads this, I would like to thank him for it.