We had a problem report outstanding for a long time, simple little problem. When a user gets a message in their inbox, a yellow marker shows up on their screen to indicate they have a message waiting. Some messages are important messages that the user has to action and then they get a red marker on their screen. Some users get the important messages as a FYI, but were still getting the red marker when they can't action the message and in this circumstance they should receive neither a red or yellow marker. Low severity and low priority bug.

Fix comes in from Development. The fix is to remove the marker for these users, which is a simple one line fix.

We test it. It is a low severity/low priority bug amongst all the other bug fixes we have to retest, to do a lot of testing in this area requires a lot of data set-up, so we just do a quick check. All is well.

The release goes to Live.

Everyone reading this knows what is going to happen next, otherwise there would be no point to this post! Yes, something was broken. True the users who were supposed not get their marker didn't get their marker, however, no other users were getting any marker either and this marker is relied on for important messages, as the users need to take action - no action means the user is prevented from using the system. Not getting the marker meant some users were not actioning and getting locked out of the system.

Big “oops”.

OK, so all test groups make this type of error sooner or later and we learn from them (sort of, sometimes we don't). But the whole episode made me think about Michael Bolton's distinction between “checking” and “testing”. What we tend to do on low severity bug retests are “checks”. Heck, we have already tested this area once, lets just check the fix. We might do some regression testing, maybe through running an automated suite, but we rarely test a bug fix that is low severity.. Maybe other test groups do more testing on this type of bug fix, but mostly my group doesn't.

So, I asked myself, SHOULD we have done more testing? In order to answer that, I looked back at all the low severity bugs we had found, over 3000 of them. I looked at how many of these fixes had either failed retest and/or caused a problem in Live. Few had failed retest, less than 1 percent. I could only find one other low severity problem that had caused a problem in Live and that was also low severity. I also estimated what would it have taken to test these fixes rather than check them? A lot was the answer. The rate of return for putting testing effort into these bug fixes would have been incredibly low.

So, my conclusion was that, yes it was bad that the bug got through, but I don't want to start testing low severity bugs, it will cost too much effort that would be better targeted at more important areas finding more important bugs. I will just have to take the risk that another low severity bug fix will cause a problem in Live.

Testing is all about assessing risks and acting accordingly.

Last week I attended the BCS SIGIST (no, that isn’t the bad thing!).  As usual, the day was well spent and enjoyable – Michael Bolton was excellent, as expected.

 

One of the presenters was Lloyd Roden.  Lloyd is a very good speaker, I have heard  speak him a few times and I like to listen to him.  Lloyd belongs to Grove Consultants and I have enormous respect for Grove and the people in it, having recommended them to a number of people and I will continue to recommend them.  His presentation this time was on his top ten controversies in Software testing and it was very entertaining, Lloyd is good at that.  I agreed with his view on 8.5 of his top ten.  The one I had a major disagreement was on certifications.  Lloyds view is that they are useful, I disagree.  That is not a bad thing that I disagree, however, I became incensed by his argument, interrupted rudely and argued with him whilst he was doing the presentation – that was the bad thing.  I should have talked to him off-line, not interrupted and argued during his presentation, especially not argued with him whilst annoyed, as I was.  Lloyd did a very good job of shutting me up politely, well done to him.

 

I have emailed Lloyd with an apology, and let this be a public apology.

 

What got me annoyed?

 

Lloyd argued that other professions use certifications, and he gave examples, therefore we should too.  He also said that ISEB/ISQTB are good because they give common terminology.  There are fallacies in this argument, as I see it, as follows:

 

Other professions do use certifications, it is true.  There is a big difference, however, between the examples given and what we have in ISEB.  The big difference is that professional certifications, like CORGI (now the Gas Safe Register), which was one of Lloyd’s examples, require the person who is certified to not only undergo training, but to show their competency.  They can also have their work inspected to show that they do good, safe, work.  From my own life, as I have mentioned before in this blog, I enjoy Ballroom and Latin American dancing (see http://www.sqablogs.com/petenairn/783/Dancing+is+like+Testing.html).  I am currently training to be a Dance Teacher.  I will (hopefully) get certified and to get that certification, I need to know the theory but also to demonstrate that I can actually dance as part of getting my certification.  I cannot even study to become a teacher until I have passed other tests that show I can dance to a reasonable level.  These are measures of competency and knowledge

 

What do you have to do with ISEB?  Turn up to a testing station, answer some multiple choice questions and hey presto, you are a certified tester!  Yes, you can go on a training course and even if you don’t then you will have to do some self study.  But, you do not have to show you can test.  This certification is, therefore, a measure of knowledge, not competency.  See my previous entry for where the two can become confused http://www.sqablogs.com/petenairn/2296/Certifications+don%26%2339%3Bt+give+competency.html

 

To be fair to Lloyd, he made it clear that you can be a good tester with no certification and you can be a bad tester and have certification, which I agree with.  So, I have to ask, what is the value of having the certification in that case?  I find it difficult to respect a certificate if someone who has never tested software can pass the exam – and I personally know of two people who have done just that.  If I know of two people, how many other people have also passed the exam who have never tested? How is that certificate valuable? 

 

Is there a way of measuring the competency of a tester by the use of an independent body?  This is where it is very difficult for a national or international certificate to be devised.  I don’t have an answer to it, except to say that when I was at IBM, we had an internal certification program whereby you had to show competency before getting that certificate.  It worked reasonably well, I was on the board that examined testers request for certification and we were very stringent in making sure we understood whether the person was competent enough to get that certificate.  Certification wasn’t done by the use of a test, but by examining the actual work that the tester had done on one or more projects.  That has its flaws too and I am not convinced in my own mind that the flaws outweighed the benefits, but it is a better measure than only sitting a test.

 

The real problem with the certification of testers, is the view that companies and recruiters take with respect to the certificate.  I see a number of job adverts with certification as a requirement to even apply.  This tells me that that the industry ascribes more weight to the certificate than it deserves. 

 

Being able to apply for jobs is the only reason I got the certificate! Does that make me a hypocrite?  Possibly, probably, but feeding my family is more important than principles to me.  I wonder how many other testers took the test for the same reason?  Is that a good reason for having a certificate? It appears to me to be the only valid reason I can find, but it isn’t a good reason.

 

Turning to the point about common terminology, which is another argument for doing certification, this is also fallacious.  Common terms exist, but what people mean by them is different in the context of the company, project, test manager involved, culture, etc, etc.  I don’t believe we will ever all agree on what is “common” because it always varies, even if slightly, depending on what you are doing at the time.  In some professions common terminology is vital, e.g. if one surgeon calls an organ a “heart”, it is useful if the assistants on the operating table know what is meant.  If we were all doing the same project at the same time, then common terminology would be important, but software development isn’t like that.

 

One other point I would like to make is that there is a difference between getting the certification and getting the training.  I have heard the argument that certifications are a good way for consultancies and training companies to make money out of training courses.  This may be true for some organisations where they only teach people to pass the exam.  Some organisations, however, provide education on testing and also teach people how to pass the exam.  The former are doing the industry no good, the latter are definitely performing a good service.  I put Grove in the latter category – I have not been on one of their training courses, but people I know and respect have and from what they tell me and from the course materials I have seen, the courses are good.  And, no, I do not have any affiliation to Grove whatsoever!

 

The arguments both for and against certifications can get emotional (as I did at SIGIST); I suspect the arguments will continue for some time and there will not be a successful conclusion for either viewpoint.

 

Last point from Lloyd’s presentation was the other 0.5 I disagreed with.  He stated that Test Managers should set aside time each week to test.  I think this is something we managers should do, but only if it makes sense on the project.  When I have over 50 testers to manage, my time is better spent on managing than doing.  By “better” I mean better for the project, better for the company, better for me and better for the testers – I am paid to manage, not to execute tests.  I will expand on this in a future blog entry.

 

So, once again, sorry, Lloyd, for my behaviour – I won’t do it again. Shall we agree to disagree?

There are a lot of people arguing for and against certifications for testing.  Personally, I don’t like certifications as I don’t believe they tell me anything about a tester’s ability to do the job.  Here is an example of where certifications can give a tester a false sense of security.

 

My customer has sent all of the User Acceptance Team onto the ISEB foundation course and they have all taken the exam.  Some have passed with no problem and some have failed, a couple more than once.

 

I was doing some fairly basic testing with one of the UAT testers the other day.  We were testing out a change to some data rules whereby the data was shown to the user, or not, dependent on dates and class of data.  I drove the system for a while and then he did.  When he was testing, I noticed that he was using dates that were way earlier or later than the boundary at which the rule kicked in.  I asked him why he had chosen those dates and he said that he was using Equivalence Partitioning such that any date would do.  OK (ish) except that the rule was very boundary dependent and the better test would be to chose dates on, just before and just after the boundary.  He also was testing every class of the data.  Again, I asked why and he said he needed to check every one.  Here was a good case of where equivalence partitioning made sense as the groups of classes behaved the same way.  I had some difficulty in persuading him that different values would be a more effective way of testing the changes as he had convinced himself in his own mind that what he was doing was right – after all, he had been on the course and had been trained in how to use these techniques, had passed the exam and therefore he knew what he was doing.

 

So, although the tester had understood the concepts of Boundary Value Analysis and Equivalence Partitioning he did not understand the applicability in a real world situation. 

 

It confirmed, in my mind, that the value of doing the training and taking the certification was low.

 

This is the problem with a knowledge-based exam/certification, it has no bearing on the competency of the person who has taken the exam.  Competency can be trained, but not by the use of certification training.

Should testers execute their own test cases?

 

Ask this question and you will get a barrage of differing views all with their reasons for a “yes” or “no” answer.  It seems that this is an area that testers and test managers just can’t agree on.  

 

So maybe there is something we can agree on, or at least debate, without having the poles apart yes/no extreme to the question.

 

Can we agree that test cases should be created?  No, we can’t. James Bach would argue that test cases have no place in doing testing.  If I read him correctly, he believes that formally written test cases have limited value and can actually be counter-productive to good testing and their only use is if there are legal reasons to create them (apologies, James, if I have mis-represented your view and you ever read this).  I have a lot of sympathy with that view and like to reduce (note, not eliminate) the number of test cases that my team writes to a minimum level necessary.  Key phrase there is “minimum level necessary”.  I believe there are occasions when the most effective way of testing something is by writing a test case.

 

Ok, so we can’t agree on whether we should write test cases, hmm, where to go from here?

 

Perhaps we can agree that tests need to be designed?  If we take the definition of “designed” as meaning that the tests are thought about before actually executing, then I think we can agree.  Even if you are doing Exploratory testing you are designing tests, by jottings on pads of paper, frameworks in your head or bright ideas that just pop-up as you learn more about the system you are testing.  Ad hoc tests are not designed, but I would argue that Ad hoc testing isn’t testing, it is playing around and has no place in the professional tester’s thinking.

 

So, if we can agree that tests need to be designed, then what is the implementation of that design?  

 

Before we determine that, maybe we need to look at what is the purpose of the test.  The widest view of the purpose of any test is to find a bug.  OK, we can argue about risk, control, confidence level, etc. but at the end of the day we are trying to find bugs.  But, it isn’t quite as simple as that.  From a project perspective, yes you are trying to find bugs, but there are supplementary purposes which are valuable to the test team, the project and the company.  Some examples:

 

·        Teaching a tester how to test.  The only way I know of teaching someone how to test is to get them to test.  Training courses are all very well, but they only give pointers on how to test.

·        Teaching a tester about the product.  It would be nice if every tester knew everything about a product before they started to test it.  Some testers may, but some (maybe all) will not.

·        Improving a tester’s skill set.  This may be in terms of getting the tester to use a previously unknown/unused technique; it may be in terms of getting them to lead a test effort.

 

The implementation of the test design, therefore, should be dependent on the purposes of the test. 

 

But we still do not have a full answer on how to implement the test design, because now we should consider what is the approach taken to testing?  By this I mean are you a test case shop, an Exploratory shop, unstructured or a mixture of approaches.  Your implementation of the test design will depend on your approach and your approach may change depending on the product you are testing this week.

 

When you have implemented your test design, then you need to execute the implementation.  Part (a small part) of me argues that if your design is good, then your method of execution is irrelevant, because you have done all the hard work of thinking and the rest is mechanical.  The rest (and more dominant part) of me argues that all you have done is laid a framework, no matter how good your test design is, the system under test is going to surprise you and you will not have covered everything you find you need to cover.

 

If that is true, that your test design implementation is only a framework, how do you determine who executes the test?  That decision should be made on who is the most appropriate person to execute the test, based, again, on what is the purpose of the test and the method of test design implementation.

 

No doubt in twenty years time people will still be arguing about whether the test case author should also execute the test.  Me?  I’ll stick with saying that “it depends” and take each situation on its merits.

I like IDMTDT testing, it can be a rich source of errors and can show up data integrity faults and usability problems that you might not otherwise find.  The big advantage is that you can then see how the system handles CRUD activities in unusual combinations.

 

Oh, sorry, IDMTDT stands for “I didn’t mean to do that” and is a set of tests that try to recover from when the user has done something stupid and wants to undo somewhere down the line in the Business process. 

 

A simple example:

 

User creates a new record for a new customer

User enters name/address/Post code

User then creates new accounts record for new customer

User puts credit limit on new customer account

User enters sales representatives name

User activates new customer.

 

At this point, User finds that the Post code is mistyped.  How do you recover?  If the system allows an edit of the customer field, then it is easy, but lets say, for the sake of the example, that there is no edit function for the Post code, it maybe a primary key in the database that is uneditable.  Let us also assume that deletion of customer records is not allowed as the system requirement is to keep all records for audit purposes and the design assumes that the user does not make this type of mistake. 

 

The recovery from this type of problem in a real world situation can be a nightmare (I have seen this type of scenario in a number of systems).  The testing can, however, be quite fun as you have a number of points at which you can try to recover the situation; after the new record created, after the account set up, after the sales rep entered, after activation.  Each of these scenarios will have a different set of tests and may involve exercising different functionality.

 

Planning and scripting such testing tends to be difficult as it is often the type of scenario not covered well in requirements or business scenarios (because people don’t make mistakes, do they?), so the best technique I have found is to include it in Exploratory tests.

 

It is a type of testing that can be forgotten or is done haphazardly.  It also, as a general guideline, should be done once the software is relatively stable as you are not trying to find functionality bugs, but business flow bugs.

Following my entry entitled “Are you positive about being negative”, Joe Strazzere asked what do I call the tests that we execute when we run out of time.  This is a good question and one that I should have answered in the original entry as Joe has correctly pointed out that I now don’t have a name for them.  

 

One of the reasons for not putting a tag on this type of testing is that it can cloud the thinking when determining what tests to run.  “Happy path”, for example, would indicate a certain type of test.

 

If you need some type of “tag”, the best I can come up with is MAT, which stands for “Most Appropriate Tests”.  Let me explain what I mean.

 

Whenever you are short of time for some tests you have to determine what tests you will run to make the best use of the time available. The thought processes go along the lines of:

 

• What are the risk areas of the system
• What are the areas that have spawned the most bugs recently
• What are the most business critical areas
• What are the areas that would be most embarrassing if they failed
• What has changed recently and therefore could have de-stabilised the system

 

So, the upshot is that I can’t give these tests a generic term, the answer is “it depends”, you should run your MAT suite!

 

Are you positive about being negative?

 

I don’t believe there is a distinction between Positive testing and Negative testing.  I don’t believe that there is such a test as a “Positive” or “Negative” test.

 

There, I have said it and the heavens haven’t opened up and I haven’t been hit by a thunderbolt.

 

You often hear the question “What is negative testing?” and you get various answers.

 

Some definitions I have read/heard:

 

1. Those test that are designed to return an error, using invalid input
2. Tests that are designed to cause the system to crash. 
3. Tests that are designed to make the system do something “off spec"

 

With reference to definition 1, so if I have an integer field and I put in an alpha character I expect to get an error, yes?  So what is negative about that test?  I have made a positive input and expect a positive response, i.e. an error message. 

 

With reference to definition 2, so, if I set up corrupt data I expect the system to crash, yes?  Again what is negative about that?  I have made a positive decision to try to force the system into an error condition to see how it handles it.  Where is the negativity?

 

With reference to definition 3, so, I am doing something to the system that might make it do something unusual.  Is this negative?  I have designed tests that try to force the system to do something it shouldn’t.  This is the closest that I can come to saying it is a “negative” test, but even here I am making a positive decision.

 

If one of the primary reasons we test is to find bugs, then we should be designing tests that are targeted at finding bugs.  Any means at the tester’s disposal should be used to find those bugs, use whatever techniques you have in your toolbox, whatever tools you have and you will execute “just tests”, not positive, not negative.

 

What does it matter?  My concern with classifying tests as positive or negative is that it is an artificial classification that clouds tester thinking.  There are no positive or negative tests, only tests that exercise the system looking for bugs.  I don’t understand the reason why testers want to distinguish between positive and negative, what benefit does it give?

 

I have heard people say “we are running out of time, only run your positive tests”.  But which are those?  I have a set of tests, I can’t differentiate them according to positive or negative.  What people usually mean by that statement is “Only run those tests that exercise the system the way the user will normally use it”.  OK, I can identify those tests, they are sometimes called “Happy path” tests, and we always create those, but they are a small percentage of the tests we prepare, I estimate less than 10% of tests fall into that category.  Are they positive tests?  By some definitions, yes, but what if one (or more) fails? Does that make it a negative test?  Again, does it matter?  A failure is a failure whatever classification is used for the test.

 

I don’t want a tester to go off and run a set of “positive” tests, I want him/her to execute tests that exercise the system in such a way that have a reasonable degree of probability of finding bugs.  That should be by exercising the system in the way the user will normally do it and doing other things as well.

 

I guess that I am not going to suddenly stop all the testers in the world from using “Positive” and “Negative” tags for tests, all I can do is try to stop my own testers!

 

Well, philk10 has been nagging me to write up last week’s SIGIST, so here goes.

 

Just for those who don’t know what this is, it is the British Computer Society Special Interest Group In Software Testing (heck of a mouthful, so SIGIST will do).  A one day conference is held every quarter and the quality of the speakers has improved considerably over the last few years, including some world known people.  Last week’s was well attended, especially considering there was a very important football match on that day.

 

The SIGIST this time had Johanna Rothman as the guest speaker.  She did three sessions, so she must have been really tired by the end of the day!

 

First session was entitled “Are you Hiring Yesterday’s Testers?”.  The presentation focussed on what a first class tester is and how to recognise the difference between first class and second class testers.  This is not concerned so much with the ability and skill of the individual tester, but attitude, maturity and skill of the hirer.  The following list, taken directly from the conference handouts, shows what first class testers looks like:

 

• Has a wide variety of testing skills
• Finds the most atrocious problems before release
• Developers enjoy working with the testers
• Learn about risks and other product information
• Can quantify the risk of a release
• Accurately predict necessary testing time and can explain why
• Understand product release is a business decision
• Supply enough information to predict rework and help projects complete on time

 

The following list, again taken from the handout, shows what second class testers look like:

 

• Routinely excluded from requirements or design meetings
• Resort to eavesdropping to hear information about the product
• Request for tools postponed or ignored
• Training budget significantly less that developer’s budget
• Interchangeable, i.e. they have such similar skills it doesn’t matter who you assign to which products
• Only work with developers on code artifacts because they either aren’t brought into the project early or they don’t know enough about the early phases to supply feedback
• Work all hours because the developers have so many defects to fix

 

Do you recognise any of those attributes from either list in your group?

 

Johanna went on to discuss how to create a first class group, which boiled down to making sure that the first class list above was achieved.

 

She also talked a bit about how to hire first class testers and it was a very short précis of her book “Hiring the Best Knowledge Workers, Techies and Nerds:  The Secrets and Science of Hiring Technical people”.

 

It was an entertaining presentation that I saw a number of people nodding their heads in agreement (no, they weren’t falling asleep!)

 

Johanna’s second session was a workshop entitled “How to Become a Sought-After Tester or Test Manager”.  This was a workshop designed to make you evaluate your skills and determine an action plan to improve them.  I think it would have worked well with a small group, but the group was large and it didn’t have the dynamics that a small workshop would have had.  Interesting, but not as useful as I had expected.

 

Johanna had the last session of the day entitled “Successful Software Management: 15 Lessons Learned”.  This was a superb presentation and I can’t do it justice here.  Basically, Johanna was going through what makes you successful as a Technical Manager.  The 15 lessons were:

 

1. Know what they pay you to do
2. Plan the work
3. Accept only on Number 1 priority at a time
4. Commit to projects after asking your staff
5. Hire the best people for the job
6. Preserve good teams
7. Avoid micromanaging or inflicting help
8. Treat people individually and with respect
9. Meet weekly with each person
10. Plan training time in the workweek
11. Fire people who can’t do the work
12. Emphasise results, not time
13. Admit your mistakes
14. Recognise and reward good work
15. Manage yourself

 

The presentation was peppered with her own experiences, mistakes and successes and whilst most managers know all these things, how many of us actually do ALL of them?

 

I went to two other presentations, the first was by Isabel Evans, entitled “A Balanced Scorecard Approach for Assessing Test Value and Success”.  This was interesting as I had heard of and seen the Balanced Scorecard Approach, but what Isabel has done is to adapt the approach for testing and how we can apply our metrics to show our value and success to the organisation.  I could see how it could be useful and I could also see that some of it might take a lot of time to put together, so I still have to think through how I can use what Isabel said in my own shop.  Her message is that what we report must be of use to the reader in their own language.  An example:  We are good at producing defect statistics, X raised this week, Y still open, Z closed, but what does that mean?  We should be couching our report in language that the reader is interested in, e.g. “the defect rate is such that the release will not be stable by next week”, or “the cost of rework due to the defect level is £x.  Her basic conclusions were as follows:

 

• Any organisation needs a balance of financial, customer, internal and innovation measures
• Test Teams exist to provide information
• Test Teams’ customers include the organisation, the project, the managers, the builders, the supporters, the users and customers and other measurement groups
• Our measurement and reporting help us
• To help our customers our measurement and reports need to reflect their goals, targets, indicators and concerns

 

The second presentation was by Graham Thomas, entitled “Seven key measures for Software Testing”.  This presentation took us through a horrendous example of a weekly test report that he had found, he showed the problems with it and then presented seven measures that help the weekly report become useful.  To be honest, there was nothing in the presentation I didn’t know and don’t use in one form or another, but Graham was a good speaker and made it entertaining.  His main focus was on showing test progress, script progress, risk profile, risk mitigation, Fault S-curve, Environment availability and Coverage.

 

The were other sessions I didn’t go to, because they clashed with the ones I did go to.

 

Conclusion: A good day, I learnt a lot.  This SIGIST was very test management biased whereas other SIGISTs have been very testing biased, so it made a nice change.  Oh, and England did win the football match…

A sledge hammer to crack a nut?

 

We are about to start testing a major enhancement to the system.  Development will be delivering this enhancement in two drops, firstly the back end and a month later the client end.  This has meant that we have had to write a utility to create the packets that the client would have sent to the host in order to test it.  This Perl utility gets responses from the tester, formats the responses and saves them away to a file for processing on the host.

 

My “plan” was to then use QuickTest Pro to be able to create many of these files and files with many packets, using a spreadsheet to drive the QTP script.  I estimated that it would take about a week to get this working as I wanted it.  Unfortunately all my QTP resources were busy on other things and could not get to it.  Panic was starting to set in until I was in the shower this morning and had a “Eureka” moment – or more likely I realised I was being stupid!  I could do the self same thing with a small VB script in Excel.  When I got into work, I spent hour and a half putting the script together, testing it and hey presto everything I needed.

 

I know a lot of people state home grown tools are often the best, but I wonder how often we plan to use the tool we have without really thinking about the best way of doing it?  I saved my team a week’s work and got something for free – well, I am a Manager, so my time doesn’t count J.

Selling Exploratory Testing

 

One of the difficulties I expected to have when introducing Exploratory Testing was having to sell Exploratory Testing to the customer and to my own management.  What I didn’t expect was having to sell it to my own testers.  Some of them had used ET before, some had not.  Those that had needed no persuading, they already knew the benefits that ET gave.  Some of those that had not used it before were very sceptical and could not see how you could test something without scripting it first.  

 

This surprised me, as testers, by their very nature, explore the system under test whether they are scripting or not (see James Bach’s articles on this on www.satisfice.com).  I tried to determine why there was this resistance to a way of testing.  The answers I came up with are, inevitably, based on a very small sample of testers (c40) and are, therefore, just my view on what I saw.  [Perhaps someone will want to pay me for a larger study J]

 

1) The way a lot of us have been trained in IT has been to document first, then do.  Just “doing” is considered to be unprofessional and cavalier, akin to hacking.  So just testing can have an uncomfortable feel to someone trained that way.  

 

2) There is the attitude that if you can’t see what I have done, then you can’t measure how good I am.  This attitude stems from the CYA attitude that comes from bitter experience that if something goes wrong, you had better have some evidence to show that you did your job.

 

3) In my own shop, we had the customer crawling all over us in the early days, distrust was rife, so everything had to be specified minutely and thoroughly reviewed by the customer.  It was drummed into the test team that documentation was everything.  This legacy meant that changing the way of working was difficult purely because it was a change and some people don’t like change.

 

4) Related to point 3, there is a fear of the unknown.  Testers who had never done ET before were nervous of failing when doing something new.  Even though they were experienced testers, it was an approach that they had to learn.

 

How did I get round this resistance?  

 

Myself and one of my Test Team Leaders who was experience in ET did a number of courses to the teams on what ET is, what the benefits are and how we were going to implement it on our project.  As part of the course, the team had to do some ET.  This ET was on a piece of the system that everyone knew, as it is core to the system and supposedly very stable.  Each of the courses found problems in this part of the system that we didn’t know were there.  None of the problems were serious, none would have caused us to stop shipping, but it the made the point that ET can find problems that had not been found.

 

We also put in place Session Based Testing (as defined by Jonathan Bach) and this helped, because the testers could see that there is documentation associated with ET and that the documentation was useful, even if it is totally different than what they were used to.

 

Now ET is a major weapon in our armoury and you wonder how we ever got along without doing it in the past.

 

Implementation of Test Ideas.

 

INTRODUCTION

 

At the British Computer Society Special Interest Group in Software Testing last year, I heard Robert Sabourin (www.amibug.com)  talk about Test Ideas and how to workshop them.  I am sure many test teams use a brainstorming approach to determining what to test, but Robert has a structured approach to brainstorming which looked interesting and I decided I would give it a go in my own team.  I have now been running these workshops for some months so I thought I would give some feedback and recommendations from our experience of using them.

 

Test Ideas workshops are a form of brainstorming that helps to structure the thinking of the testers and allows fairly easy documentation of the test ideas.  The Following section is my précis and downright plagiarism of Robert’s article “What Not to test” a link to which is included below (although I just tried the link and it didn’t work).  This document is what I used to provide guidelines to my test team.

 

TEST IDEAS

 

A test idea is a description of a kind of test that needs to be run. These are documented in a table as follows:

 

ID	Source	Description	Size	Importance
			Small, Medium, Large	Low, Medium, High

 

The fields in the tables are:

ID is just a reference number.

Source is the source of the test idea. There are many sources for these testing ideas; some are shown here:

Testing Idea Source	Comment
Requirements (REQ)	What should the system do?  What capabilities should it have?  What are its performance criteria? Under what constraints does it operate? Etc.
Failure Modes (FM)	What can break and fail?  What data can be wrong, missing or incorrectly structured?
Usage Scenarios (USAGE)	What do people do when they use the system?  How can we model operational workflow?  What other system and manual processes do they use while using the system?
Quality Factors (QF)	What characteristics must the system possess to have quality?

 

Description briefly details each idea. Notice that the ideas aren’t specific enough to be test procedures, but they are specific enough that a tester easily could flesh them out. We will invest in detailing an idea more specifically if (and only if) we decide to use it. If we defer or reject a testing idea, there is very little benefit to further detailing or elaboration. Imagine the waste of spending several days detailing a test procedure only to decide not to implement the associated testing idea. In testing, size matters—hence the field size.

Size. Consider the size of a testing idea to be an estimate of the reasonable amount of effort you would have to invest to satisfactorily implement that idea given the state of the software under test and the skills and competency of the testing team. Small is a testing idea that takes less than ninety minutes to implement. Medium is a testing idea that takes a day to implement, and large is a testing idea that takes about a week to implement. If test ideas are larger than large, we split them or otherwise reorganize them.

Importance. The bottom line is that a test idea is not important if the bug it finds will not be fixed. For example, if we decide not to fix spelling mistakes in final system testing, then any test idea related to spell checking is unimportant. A testing idea is important when the information it provides contributes to the decision to deploy or ship the software. High implies knowledge of the test passing was critical and low implies a decision to ship or deploy the software could be made without knowledge of the test result. Anything in between is considered medium.

 

CAPTURE TESTING IDEAS

To fill in the table, a structured brainstorming session is held  which is designed to capture testing ideas. We can decide not to implement a testing idea based on the information we have now, but if new information comes our way in the future, we can reconsider using the testing idea. There are no wrong testing ideas. They all have some value even if they are not implemented.

 

 

Test Idea Triage

Test ideas are not static. As testing progresses, new ideas are discovered. As testers and developers start using the application from an end-user perspective, usage, scenario, and inter-operation testing ideas are generated.

 

Further Information

 

See http://www.ece.mcgill.ca/~info429/2004_10_07_What_not_to_Test.pdf

 

My team’s experience

 

Test Ideas Workshop

 

Everyone in the workshop is given index cards and is given 20 minutes to write down at least 4 ideas against each type of test.

Once complete, the cards are collected up and posted on flip charts and the moderator reads them out, asks for clarification if the idea is unclear, but no discussion is allowed at this point.

Once read out, the floor is open for further ideas.  These ideas are written up on the flip charts as they occur.  Discussion is allowed at this point, not on the merits of the ideas but on further possibilities and ideas.  This sparks off further ideas which are also captured.

When the workshop has run out of ideas, the moderator runs though each idea and the workshop agrees Importance and size and eliminates the duplicates.  Typically, this also generates some new ideas and these are also captured.

Finally, four “volunteers” are assigned to write up each of the sets of flip charts in the standard format.

 

Our Method

 

We took the Robert’s method and decided to apply it as Robert stated, with two exceptions:

 

·         Robert says that you write one idea on one index card (he has pre-printed cards).  We decided this was too messy when you had a number of people in the workshop, so we allowed more than one idea on one card.

·         Robert suggests that the Size and Importance is determined by the person writing the idea, we decided that we would not do that, we would determine the size and importance as a group at the end so that we were not arguing about size or importance when we should be coming up with ideas.

 

What was good?

 

It was abundantly clear from the very first workshop that this method beat brainstorming hands down.  The number of ideas was greater and the quality of those ideas was superior. 

The team in the workshop were enthused about the project and got their focus on the testing much earlier than it would have been otherwise.

You have a very good starting point for doing your test preparation

 

What do you need to beware of?

 

As with all types of workshops or brainstorming sessions, you need to be aware of the dominant personality problem.  This is where the 20 minute session at the beginning helps a great deal because even the quieter members of the team have a say right at the start, getting them involved.

 

You need a good moderator who can control the workshop.  Everyone talking at once, arguments, sub groups forming are all bad for the overall workshop and the moderator needs to be strong enough to prevent the bad things happening.

Time flies!  We scheduled 1.5 hours for a workshop that took over 3 hours.  It will always take you longer than you think.

 

Make sure that the ideas are written up as soon as possible after the workshop so that everyone can see them.  This keeps the momentum going and allows people to critique the write-up.

 

 

Conclusions

 

These workshops work well.  They give the testing a great kick start.  

You can use the test ideas in a number of ways, you can produce manual scripts from them, automated scripts from them, you can use each idea or a group of ideas as session objectives for Exploratory Testing.

 

I would highly recommend this method and, if Robert Sabourin ever reads this, I would like to thank him for it.

Regression Testing algorithm

 

When a large system has been developed and gone live, there are, inevitably, maintenance releases, how do you determine what to regression test?  For example, on my current project we went live 12 months ago and we are putting out maintenance releases every 6 weeks.  These maintenance releases are not purely live bug fixes, in fact only a small number of live bugs have been found.  The majority of the maintenance release contain changes that the customer wants.  These changes have been sitting around for some time as the customer wanted the core system to be live before implementing any changes. 

 

Evidence during previous regression cycles has shown that a number of functions have displayed faults that were not in the previous versions of the code.  This may be as a result of the changes breaking existing functions or bug fixes breaking functions or merged code causing faults or incorrect version control or other reasons.  One of the difficulties with a large, complex, system is that a change in one part of the system can affect another part without anyone realising it.  All the analysis in the world cannot stop this type of knock-on affect from happening.

 

This indicates that there is an ongoing need for regression testing of these functions.

It can be expected that a number of live bugs will be raised as a result of continued live running, so these have the potential to adversely impact existing functionality.

In order to perform a full regression test on the system it would take approximately 3 months for the entire team, I did say this is large and complex.  Automation helps, but the system is such that large parts of it are either very difficult or impossible to automate.

 

Given the timescales for System Testing as part of a maintenance release it is not, therefore, practical to perform a full regression test, so the strategy is to perform what I call a rolling regression test.

The approach to regression testing at each maintenance release is as follows:

·         Any function affected by a fixed bug or a change will be regression tested

·         A subset of non-affected functions will be regression tested

·         The Automated Regression Suite will be run at least once.

 

The “non-affected” functions are those that appear not to have been impacted by any change happening in the release.  With over 400 functions in the system, each release should only affect a small number of these functions. (yeah, right!).

 

The algorithm for determining the subset of non-affected functions that will be regression tested is as follows (forgive the pseudo-code, it was the best way I could think of to get the message across):

 

BEGIN REGRESSION TEST ALGORITHM

Available regression test time = Total team time – (Time needed for Change testing + Time needed for Bug fix retesting)

FOR each function

                        SELECT

                        CASE been affected by Bug fix or Change

                                    Add to Regression Candidate List 1

                        CASE been regression tested in the last released

                                    No regression

                        CASE Business Critical Function

                                    Add to Regression Candidate List 2

                        CASE ELSE

                                    Add to Regression Candidate List 3

                        END-SELECT

END-FOR

FOR List = Regression Candidate List 1, Regression Candidate List 2, Regression Candidate List 3

                        FOR each function in List

                                    IF time to regression test this function < Available regression test time

                                    THEN

                                                Regression test the function

                                                Available regression test time = Available regression test time – time to regression test this function

                                    END-IF

                        END-FOR

END-FOR

END REGRESSION TEST ALGORITHM

 

This algorithm will, therefore, cycle round the functions to ensure as broad a coverage as possible on regression testing functions.  Over a number of maintenance releases, every single function will have been regression tested at least once.