How do you know when you are “getting it right” with your customer?

 

I have dealt with customers as both Test Manager and Project Manager for some years now and I used to worry about whether I was “getting it right”.  I came to the conclusion some time ago that if you aren’t getting complaints and your management are not getting complaints about how you work with them, you must be doing alright.  Here is a little story about something that happened this week that made me feel really good about my relationship with the current customer.

 

Background:

 

I started on this project 3.5 years ago and the Test Group relationship with the customer was one of the worst I have ever come across.  They didn’t trust us, accused us of lying, hiding facts and the customer meetings I had were extremely difficult.  I had to work really hard to improve that relationship, which I think I managed to do.  One of the things I did which helped was to instigate a weekly bug review with the customer.  At that bug review, we review all the bugs that have been raised in the week (from System Test and Live) and all the bugs that have changed to a terminal state (i.e. closed or set to “not a fault”).  The purpose of this meeting was to make sure we all agreed whether it is a bug, that the severity is correct and the schedule for fixing is reasonable.  This meeting has been running for just under 3 years and has been very successful in improving openness and mutual understanding.  

 

One of the things that happens to us in software development is that a fix or enhancement can break something else or that a bug re-appears for whatever reason.  The UAT Manager at these meetings christened such bugs as coming from the “Change Fairy”.  A number of bugs over the years have been classified as having occurred due to the Change Fairy.

 

The story:

 

This week we held a “Celebration of Success” dinner which was a dinner for a number of the project team, customer and suppliers, about 100 people in all.  It was a posh do, dinner jackets, very formal.  After we all had eaten, there were presentations to a few people and speeches.  The UAT Manager presented to me a “Change Fairy”.  This was something he had made himself out of bits he had got from an old clock, a fairy that he had put on some wings, put into a cage and put on a plaque with the motto “Change Fairy – Do not Release”.

 

I think this shows how much our relationship has improved and is now very good, that a customer who had totally distrusted us can now pull a comical stunt like that at a public function.  I was struck dumb and so pleased. 

 

Yes, I think I am “getting it right” with this customer.

 

 

This item has come about following reading this item in qaforums http://www.qaforums.com/ultimatebb.php?ubb=get_topic;f=46;t=002204;p=3 where there was a brief exchange between Jim Hazen and myself on the subject of loyalty.  Loyalty is important to me, it is important as I believe that loyalty is an important part of working life.  I accept that it is important to me and some people do not consider it to be as important and that is fine, we all have different things that are important to us.

 

Firstly, here is a definition of loyalty: “Loyalty, as a general term, signifies a person’s devotion or sentiment of attachment to a particular object, which may be another person or group of persons, an ideal, a duty, or a cause. It expresses itself in both thought and action and strives for the identification of the interests of the loyal person with those of the object”

 

I get annoyed with people who expect loyalty.  You find it amongst some managers who believe that because they have some title that they not only deserve loyalty, it is their right to be given loyalty.  You also find it amongst the “experts” who believe that no-one can tell them anything.  This arrogance annoys me and I am much less likely to be loyal.  That is my personal reaction which other people may or may not consider to be a good reaction!

 

What everyone should remember is that loyalty is a two way street; I am only loyal to someone who is loyal to me.  You have to earn loyalty from your peers, your managers and from your team, you are never given it.  You can’t demand loyalty, it is always earned.

 

So, people are loyal to other people and to companies.  Companies are loyal to their staff and other companies.  The level of that loyalty varies widely depending on a number of factors, which I am going to discuss further.

 

What makes someone loyal to a person or a company is dependent on the person themselves, what “pushes their button”.  Some people are loyal because they have a nice office/cube/desk, some because they have good perks, some because they are paid well.  This is what I call doggie loyalty.  I have a dog, he is a 12 year old cross-breed terrier and he is incredibly loyal.  He would do anything for me, but if I stopped feeding him, how long would that loyalty last?  Not long.  Human beings are not dogs, however, if you stop feeding them what they want (money, benefits, etc.) their loyalty will stop.  Some people will not be loyal even if you do feed them, these people require more.  They require someone or something to give them a purpose in life, like success, acknowledgement, position, which could be called social loyalty.  If you look at Maslow’s hierarchy of needs http://www.businessballs.com/maslow.htm, then you can see what it is that breeds loyalty above the doggie loyalty, or level 1 and 2 in Maslow’s hierarchy.

If you look around the people who work with you, what level of loyalty do they display?  Is it doggie loyalty, or is it social loyalty?  What level of loyalty do you display?

 

It is my experience that there are a number of people who only have doggie loyalty to a company, these are the people who will change company at the drop of a hat if they are offered more money elsewhere, contractors are more likely to show this type of loyalty although some permanent staff also fall into this category.

 

If you show a certain level of loyalty to a company or person, that is the level of loyalty you should expect to receive.  If you only display doggie loyalty, then why should you expect to be given social loyalty?  If you display social loyalty and only receive doggie loyalty (or less), then you are either entitled to be disgruntled or you should re-evaluate the level of loyalty you are prepared to give.

 

I posit that (although I know of no study to back this up) if you show loyalty to your company, they will show loyalty to you. A company may display doggie loyalty, i.e. it is only interested in the profits it can make.  Some are more altruistic and care about their employees.  A doggie company will not care how loyal you are, but a more employee friendly company will reward loyalty with loyalty – again, you have to earn it by your display of loyalty.

 

If you are manager or team leader, you should want to inspire loyalty in your team(s).  Loyal team members work more effectively.  Loyalty means that you will get more out of them, they are more willing to do extra if asked of them.  Determine what it is that gets you that loyalty in every individual and do what you can to give it to that individual.  I strive first for personal loyalty as I believe that loyalty to me is the first step in gaining loyalty to the company.  I don’t always succeed, some people will never be as loyal as I would like, but often I do succeed.

 

Something to beware of, as a manager, don’t confuse friendship with loyalty.  It is very difficult (for me, at least) for a manager to be a friend and a manager of anyone who is working in your team.

 

So, to sum up, loyalty breeds loyalty and if you display it, you will often receive it.  If you display doggie loyalty, then you should only expect to receive doggie loyalty.

 

Agile Gardening

 

I am a keen gardener, I love pottering around the plot and adding plants here, moving plants there, weeding, feeding, watering and then just sitting and looking at it.  Now and again, I get the urge to do something a bit more and redesign part of my garden.  I started redesigning my back garden this year.  I thought about which GDLC (Garden Development Life Cycle) I would use and it came down to a choice of the Waterfall (no, not lots of water features) model or using Agile methods.  I decided on the Agile methods.  I have previously sat down with garden design software and designed my garden in detail, showing the hard landscaping, the shape of the lawn, exactly which plants would go where and using the clever software features of showing what the garden would look like in 10 years.  There is an element of satisfaction when you then print this plan on A3 and pour over it, tweaking bits before finalising the plan.  Then you get a contractor to quote for the work you want doing (I never have the time to do it all myself), you get a timescale, you plan out the deliveries of plants and hard material, work out which plants you are going to keep and where they need to be kept whilst the work is going on.  The work starts and is inevitably late finishing, hampered by the weather, unreliable deliveries, faults in the plan needing rework and the whole project becomes frustrating.  The enjoyment of having a new garden evaporates because of the headaches to get it done and for some reason it never quite looks as good as you had planned it.

 

With Agile gardening, it is much more fun.  I had a broad view of what I wanted, I wanted to screen off my shed and greenhouse, replace the conifer hedge with a fence and make a back border, move the patio from the edge of the garden to the centre and make a woodland garden where the patio used to be.  So, the parts of the product are Back Border, Patio, Woodland Garden and Greenhouse Screen.  I got the landscape company in to broadly quote for the work, within the budget I have and they were very happy to have a broad brief that enabled them to use their creativity to produce a good garden.  As the work started, we discuss the next step at short regular meetings, what had worked so far and what we needed to change and the garden started to take shape.  A problem tracking system has been put in place so that we can record what needs changing (I call it GardenZilla)

 

Of course, I did some work myself.  This was in the form of some Exploratory Digging (or ED). I wanted to see whether the border at the back where the hedge had been replaced would stand some deep rooted shrubs.  I dug that border in a series of Exploratory Sessions and the debriefs with the boss (my wife) happened every 2 hours over a cup of tea or lunch so that we could discuss progress and what further Exploratory Sessions could be done.  I kept learning about the depth of the soil along the border and was able to determine where I needed to add more organic material and where there was enough so I could work out which plants went where.  Having done the ED, I then went on to the Exploratory Planting (EP).  Here all the plants I had bought, grown from seeds or cuttings or saved when other parts of the garden were dug up were placed in what I thought were the best places for them.  I then looked at them and moved a few around.  I then went indoors and looked from the upstairs window to see what the effect was.  This caused some more movement of plants.  Once happy, they were all planted in the border.

 

The back border is now Live and so far there are no problems, but as the summer approaches, I expect there to be some mistakes so they will be added to GardenZilla.

 

The garden is not yet fully complete.  Back Border is Live, Patio is work in progress, Woodland area and Greenhouse Screen are not started,

 

There is some excitement in seeing the broad view coming together, the landscaper is happy, I am happy and the end product looks like it will be good.

 

As each part of the garden goes Live, we can establish the goals of the next part of the garden and work towards them.

 

I now need to ask myself am I bringing my work home or am I taking my hobby to work?

There are a number of ways of measuring the processes in a test organisation, CMMI, TPI being just two. It is all very well having these methods and any way of measuring is useful, but how do you go about determining where your problems are and how to resolve them?  TPI is great and it does provide some useful pointers, but what are you going to do in your shop?

 

A method I have found useful is a SWOT analysis.  For  anyone who does not know , SWOT stands for Strengths, Weaknesses, Opportunities, Threats. 

 

Strengths are those things you as a test team are good at.

Weaknesses are those things that you as a test team are not good at.

Opportunities are those things that are outside of the test team that if happened would be good for you

Threats are those things that are outside of the test team that if happened would be bad for you.

 

I have run this exercise a number of times in order to improve the processes that we use in test teams and it has proved quite successful.

 

You run the SWOT analysis as a workshop, or series of workshops and you will invite some of your key team members to participate.

 

The last time I ran the exercise at the end of 2005, I split the workshops into the following subjects:

 

• Test Preparation
• Test Execution
• Interfaces and communication
• Bug process
• Test Management
• Releases to Testing

The first set of workshops identified the SWOT for each of these subjects and then the second set of workshops identified the actions that we could take, who would take the actions and when they should be done by.

 

The actions were then monitored over a period to ensure that they got done.

 

It is difficult to measure the effect of each of the individual actions, but following the workshops, I can see that we are getting more effective and better at what we do.  The exercise will be done again soon, so that we should get a new set of actions to continually improve.

 

An example of the output follows:

 

Test Process under examination:  Bug Process

 

Strength:  Good Level of detail in bug reports

Weakness: Analysis of No Fault Founds take too long

Opportunities: Fix fail rate is improving

Threat: Bug reports from Help Desk have insufficient detail making them difficult to retest

 

Actions: Educate Help Desk into better bug reporting.  Analyse fix fail rate to improve further.  Work closer with developer on No Fault Founds,.

 

 

On the last exercise, we identified 66 action points that would make us more effective and/or efficient.  Most of these actions have now been completed and we will soon be ready to go again. 

 

Implementation of Test Ideas.

 

INTRODUCTION

 

At the British Computer Society Special Interest Group in Software Testing last year, I heard Robert Sabourin (www.amibug.com)  talk about Test Ideas and how to workshop them.  I am sure many test teams use a brainstorming approach to determining what to test, but Robert has a structured approach to brainstorming which looked interesting and I decided I would give it a go in my own team.  I have now been running these workshops for some months so I thought I would give some feedback and recommendations from our experience of using them.

 

Test Ideas workshops are a form of brainstorming that helps to structure the thinking of the testers and allows fairly easy documentation of the test ideas.  The Following section is my précis and downright plagiarism of Robert’s article “What Not to test” a link to which is included below (although I just tried the link and it didn’t work).  This document is what I used to provide guidelines to my test team.

 

TEST IDEAS

 

A test idea is a description of a kind of test that needs to be run. These are documented in a table as follows:

 

ID	Source	Description	Size	Importance
			Small, Medium, Large	Low, Medium, High

 

The fields in the tables are:

ID is just a reference number.

Source is the source of the test idea. There are many sources for these testing ideas; some are shown here:

Testing Idea Source	Comment
Requirements (REQ)	What should the system do?  What capabilities should it have?  What are its performance criteria? Under what constraints does it operate? Etc.
Failure Modes (FM)	What can break and fail?  What data can be wrong, missing or incorrectly structured?
Usage Scenarios (USAGE)	What do people do when they use the system?  How can we model operational workflow?  What other system and manual processes do they use while using the system?
Quality Factors (QF)	What characteristics must the system possess to have quality?

 

Description briefly details each idea. Notice that the ideas aren’t specific enough to be test procedures, but they are specific enough that a tester easily could flesh them out. We will invest in detailing an idea more specifically if (and only if) we decide to use it. If we defer or reject a testing idea, there is very little benefit to further detailing or elaboration. Imagine the waste of spending several days detailing a test procedure only to decide not to implement the associated testing idea. In testing, size matters—hence the field size.

Size. Consider the size of a testing idea to be an estimate of the reasonable amount of effort you would have to invest to satisfactorily implement that idea given the state of the software under test and the skills and competency of the testing team. Small is a testing idea that takes less than ninety minutes to implement. Medium is a testing idea that takes a day to implement, and large is a testing idea that takes about a week to implement. If test ideas are larger than large, we split them or otherwise reorganize them.

Importance. The bottom line is that a test idea is not important if the bug it finds will not be fixed. For example, if we decide not to fix spelling mistakes in final system testing, then any test idea related to spell checking is unimportant. A testing idea is important when the information it provides contributes to the decision to deploy or ship the software. High implies knowledge of the test passing was critical and low implies a decision to ship or deploy the software could be made without knowledge of the test result. Anything in between is considered medium.

 

CAPTURE TESTING IDEAS

To fill in the table, a structured brainstorming session is held  which is designed to capture testing ideas. We can decide not to implement a testing idea based on the information we have now, but if new information comes our way in the future, we can reconsider using the testing idea. There are no wrong testing ideas. They all have some value even if they are not implemented.

 

 

Test Idea Triage

Test ideas are not static. As testing progresses, new ideas are discovered. As testers and developers start using the application from an end-user perspective, usage, scenario, and inter-operation testing ideas are generated.

 

Further Information

 

See http://www.ece.mcgill.ca/~info429/2004_10_07_What_not_to_Test.pdf

 

My team’s experience

 

Test Ideas Workshop

 

Everyone in the workshop is given index cards and is given 20 minutes to write down at least 4 ideas against each type of test.

Once complete, the cards are collected up and posted on flip charts and the moderator reads them out, asks for clarification if the idea is unclear, but no discussion is allowed at this point.

Once read out, the floor is open for further ideas.  These ideas are written up on the flip charts as they occur.  Discussion is allowed at this point, not on the merits of the ideas but on further possibilities and ideas.  This sparks off further ideas which are also captured.

When the workshop has run out of ideas, the moderator runs though each idea and the workshop agrees Importance and size and eliminates the duplicates.  Typically, this also generates some new ideas and these are also captured.

Finally, four “volunteers” are assigned to write up each of the sets of flip charts in the standard format.

 

Our Method

 

We took the Robert’s method and decided to apply it as Robert stated, with two exceptions:

 

·         Robert says that you write one idea on one index card (he has pre-printed cards).  We decided this was too messy when you had a number of people in the workshop, so we allowed more than one idea on one card.

·         Robert suggests that the Size and Importance is determined by the person writing the idea, we decided that we would not do that, we would determine the size and importance as a group at the end so that we were not arguing about size or importance when we should be coming up with ideas.

 

What was good?

 

It was abundantly clear from the very first workshop that this method beat brainstorming hands down.  The number of ideas was greater and the quality of those ideas was superior. 

The team in the workshop were enthused about the project and got their focus on the testing much earlier than it would have been otherwise.

You have a very good starting point for doing your test preparation

 

What do you need to beware of?

 

As with all types of workshops or brainstorming sessions, you need to be aware of the dominant personality problem.  This is where the 20 minute session at the beginning helps a great deal because even the quieter members of the team have a say right at the start, getting them involved.

 

You need a good moderator who can control the workshop.  Everyone talking at once, arguments, sub groups forming are all bad for the overall workshop and the moderator needs to be strong enough to prevent the bad things happening.

Time flies!  We scheduled 1.5 hours for a workshop that took over 3 hours.  It will always take you longer than you think.

 

Make sure that the ideas are written up as soon as possible after the workshop so that everyone can see them.  This keeps the momentum going and allows people to critique the write-up.

 

 

Conclusions

 

These workshops work well.  They give the testing a great kick start.  

You can use the test ideas in a number of ways, you can produce manual scripts from them, automated scripts from them, you can use each idea or a group of ideas as session objectives for Exploratory Testing.

 

I would highly recommend this method and, if Robert Sabourin ever reads this, I would like to thank him for it.

Using a Test Project Office

 

This entry was prompted by this thread in QA forums.  http://www.qaforums.com/ultimatebb.php?ubb=get_topic;f=18;t=001356

 

 

Test Project Office staff are, in my experience, under-used resources that if used properly can assist in making the running of a Test team much more efficient and effective.  Until very recently, I was running a test team of 45 testers and had my own Test Project Office.  The Test Project Office consisted of an experienced Project Manager and two admin staff.  It was the first time I had a Test Project Office of my own, I had always used the Programme Office or overall Project Office before, and it took some time for me to realise how a Test Project Office can make my life easier.

 

Test Project planning is a key area that the Test Project Office can take a load off the Test Manager.  It is my responsibility to have a realistic plan that reflects the Test Project, but do I have to actually do it myself?  As a related aside, let’s have two definitions that so many people confuse. 

 

·         Test Plan – A test plan consists of details of what is to be produced, how it will be produced, what resources are required (both human and non-human), what timescales will be set, what are the key deliverables, what are the success criteria, entry and exit criteria, communications paths, responsibilities, project controls, external dependencies and who will deliver them, and usually a number of other bits of information. 

·         Test Schedule – a Gantt chart showing a work package breakdown, when each task will be started, finished, who is doing that task, dependencies between tasks.

 

A Test Plan must contain a schedule, but thinking that a schedule is a plan is a common mistake.  Part of the problem is the tools used.  A common tool is MS Project and it is misnamed, it should be called MS Schedule because that is was it does, it doesn’t plan.

 

Only managing the schedule is doing half the job, the whole plan needs to be managed.  If you have an experienced Project Manager assisting you, then s/he is perfectly capable of managing and maintaining your plan so you don’t have to. 

 

As a Test Manager, the plan is only a part of what you need to worry about.  The most important part of the job, in my opinion, is making decisions.  You make decisions based on the data and information that you discover and is provided to you.  The less data gathering and analysis you have to do, the more you can concentrate on making the right decisions in a timely manner.  Here is where the Test Project Office can assist. 

 

What are the main pieces of information I need in order to make decisions?  Here is a list:

 

·         Delivery dates to me

·         Progress against Development schedule

·         Progress against Test schedule

·         Reasons for dates changing

·         Problems that may impact my plan (risks)

·         Problems that will impact my plan (issues)

·         Bug raising rate

·         Bug fixing rate

·         Actual deliveries

 

All of these are activities that I don’t need to do myself, what I need are the output from these activities. 

 

The decisions I can then make are:

 

·         What should I do to mitigate the risks

·         What can I do to resolve the issues

·         How should I juggle my resources to attempt to make the dates without compromising the testing

·         How can I use my resources more effectively

·         Do we need to re-scope the project

·         Do we need to re-schedule the project

·         Who do I need to kick into action

 

The decision making activity is what I believe is the added value I bring to the project, not the gathering of data.  Hence, the Test Project Office is a very valuable asset in helping me to do that.  Their key activities are:

 

·         Maintenance of the schedule – Here I want them to gather the actuals, identify slippages and early completions, identify over and under-resourcing, do what-if analysis on the schedule to provide me with options.  I also want new or removed activities to be folded into the schedule to identify the impact of the change so I can make a decision.

·         Maintenance of the plan – Here I need to ensure that the plan stays current, that the statements made are still true and any changes are correctly communicated and agreed.  The Test Project Office can do that for me.

·         Bug analysis – It would be nice if bug analysis always followed the S-curve, but the reality is that it doesn’t always.  Regular plotting of bug raise rates, fix rates and retest rates, by severity and priority will give information about whether we are in trouble as a project or not.  Ad hoc analysis is often required, for example I may want to know which function has had the most number of bugs raised against it or what is the level of No Fault Found bug reports.  Again, this information allows me to make decisions.

·         Admin – with a large team goes a lot of admin.  I don’t want to do it myself, apart from the fact I am not good at it, it is a waste of my time

·         Advice and guidance.  Having an experienced Project Manager looking after the Test Project Office for me was extremely valuable.  He was able to provide me with advice and guidance, I used him as a sounding board for ideas, he pointed out mistakes I was making or about to make and avoided some potentially embarrassing situations.

·         Rumour! – Project Office staff are often able to get information that you can’t get yourself.  Having “Test Manager” tattooed on your forehead can exclude you from some information if people are concerned about what you will do with it or if they don’t really want you to know or if the information is not final yet.  Rumour is a vital source of information that assists in making the right decisions.

 

How to use a Project Office is not something that I have ever seen taught in standard Management training courses and this is an omission.  I have written and given a Project Office training course when I was doing Project Management training courses and it is a skill that is worth gaining.

 

If you don’t have your own Test Project Office, then use the Project Office set up for the Project.  If you don’t have a Project Office, then I hope you enjoy filing!

Test Management musings – Part 1

 

I set up this category just to jot down miscellaneous thoughts about test management.  I’ll see how this goes, it might seem like a bad idea after a while.  What follows are personal musings, often not based on fact, merely my opinion based on my experience and observations.

 

People are the only real asset that a Test Manager has. Yes, we have licenses for tools, we have test lab equipment but they are no use if we don’t have people.  A good part of any manager’s time is spent managing their people and Test Managers are no different.  Getting the best out of your people is key to making your team a success and having managed a number of test teams, development teams, Project Managers, Support teams, I think that testers pose a particularly different challenge.  

 

Since I started managing test teams (nearly 20 years ago) one of the things that has fascinated me is the characteristics, attitudes and behaviours that are peculiar to testers.  

 

We are, generally, an odd bunch, we see work life differently than almost anyone else in IT.  Our minds are tuned and trained to ask the “What if” questions whereas most other people in IT ask the “How can” questions.  Example:  If a requirement is phrased as “I want to grow tomatoes in February”, a developer will ask “How can we get sufficient heat and light in the English winter to grow tomatoes?”  A tester will ask “What if you get an early Spring, will mildew destroy the crop?” A totally different mindset. 

 

We are motivated by different things.  Any tester worth their salt is delighted to find a bug and the level of delight is different depending on how difficult it was to find it and how much effort we had to put in to find it.  Testers are only slightly happy to find a crash when inputting one field on the first input screen, that is too easy, but spend 2 days setting up a detailed, complicated scenario and finding a data inconsistency and the tester is over the moon and can recount this testing exploit to every other tester in the world with great enthusiasm (I exaggerate slightly!).  So, why then does the tester come to me and say “I’m sorry, I have found a serious bug in Function X” why are they sorry?  They have done their job well.  I do the same thing myself, I’ll go to a Project meeting and state, “I am sorry, we have found a serious bug and I believe it would be too risky to ship the system to live”  Why am I sorry?  My team have done great job in finding the bug and I am proud that they have, so why do I feel the need to say I am sorry?  This strange mixture of delight and expressions of guilt is a characteristic of every test team I have managed.  We are motivated by finding bugs but feel the need to hide our delight.  I don’t pretend to fully understand, although I have a few theories that I might put down in a later entry.

 

More on managing testers later (probably)

Regression Testing algorithm

 

When a large system has been developed and gone live, there are, inevitably, maintenance releases, how do you determine what to regression test?  For example, on my current project we went live 12 months ago and we are putting out maintenance releases every 6 weeks.  These maintenance releases are not purely live bug fixes, in fact only a small number of live bugs have been found.  The majority of the maintenance release contain changes that the customer wants.  These changes have been sitting around for some time as the customer wanted the core system to be live before implementing any changes. 

 

Evidence during previous regression cycles has shown that a number of functions have displayed faults that were not in the previous versions of the code.  This may be as a result of the changes breaking existing functions or bug fixes breaking functions or merged code causing faults or incorrect version control or other reasons.  One of the difficulties with a large, complex, system is that a change in one part of the system can affect another part without anyone realising it.  All the analysis in the world cannot stop this type of knock-on affect from happening.

 

This indicates that there is an ongoing need for regression testing of these functions.

It can be expected that a number of live bugs will be raised as a result of continued live running, so these have the potential to adversely impact existing functionality.

In order to perform a full regression test on the system it would take approximately 3 months for the entire team, I did say this is large and complex.  Automation helps, but the system is such that large parts of it are either very difficult or impossible to automate.

 

Given the timescales for System Testing as part of a maintenance release it is not, therefore, practical to perform a full regression test, so the strategy is to perform what I call a rolling regression test.

The approach to regression testing at each maintenance release is as follows:

·         Any function affected by a fixed bug or a change will be regression tested

·         A subset of non-affected functions will be regression tested

·         The Automated Regression Suite will be run at least once.

 

The “non-affected” functions are those that appear not to have been impacted by any change happening in the release.  With over 400 functions in the system, each release should only affect a small number of these functions. (yeah, right!).

 

The algorithm for determining the subset of non-affected functions that will be regression tested is as follows (forgive the pseudo-code, it was the best way I could think of to get the message across):

 

BEGIN REGRESSION TEST ALGORITHM

Available regression test time = Total team time – (Time needed for Change testing + Time needed for Bug fix retesting)

FOR each function

                        SELECT

                        CASE been affected by Bug fix or Change

                                    Add to Regression Candidate List 1

                        CASE been regression tested in the last released

                                    No regression

                        CASE Business Critical Function

                                    Add to Regression Candidate List 2

                        CASE ELSE

                                    Add to Regression Candidate List 3

                        END-SELECT

END-FOR

FOR List = Regression Candidate List 1, Regression Candidate List 2, Regression Candidate List 3

                        FOR each function in List

                                    IF time to regression test this function < Available regression test time

                                    THEN

                                                Regression test the function

                                                Available regression test time = Available regression test time – time to regression test this function

                                    END-IF

                        END-FOR

END-FOR

END REGRESSION TEST ALGORITHM

 

This algorithm will, therefore, cycle round the functions to ensure as broad a coverage as possible on regression testing functions.  Over a number of maintenance releases, every single function will have been regression tested at least once.

 

As people seemed to like my last story, here is my favourite story of a user doing something to break a system, which also shows how testing techniques can fail.

 

We were building a large, complex system for a customer and spent 2 years on the development and testing.  I was the Test Manager and by the end of the project I was very pleased with the way it had gone from a testing perspective.  It was one of those rare projects where the requirements had been very well specified by the customer, the specifications had been well produced, the unit testing had been good and the System Testing had been given sufficient time to do a good job.  This was an extremely important application that had serious consequences if it failed.

 

When the project went live, I was asked to be the Support Manager on the live site.  My role was to train the Operations staff in how to run the system and to provide on-site support for any problems that came about.  

 

The system ran fine, with some small hiccups, for about 3 months and we and the customer we very pleased with the way it was running (apart from one “problem” that is a subject of another user story).

 

Then, one day, the system crashed.  Big time.  Corrupt data, users couldn’t do anything, machine had its legs in the air doing a good impression of a heap of junk.  The part of the system that had failed was an interim in-memory “database” that was used to capture user input before being written to the database on disk and then onto Optical disks.  I attempted to diagnose the fault and had no idea how the problem had arisen.  The fault logs were of no use whatsoever.  I cleared the in-memory database, reset the database and started everything up again.  This took about 2 hours which was a disaster at a critical time of the day.  The users then had to re-input all their data.  I increased the logging level and left it running. 

 

All went fine for a couple of days and then it happened again.  The customer was now livid and threatening all sorts of legal action, so the pressure was on.  Again, I got the machine running after copying off the logs.  It took me a couple of hours to plough through the logs and I found the problem.  The main user input to the system was names and addresses and the data held in interim memory was just a stream of data with the field separator of a double quote.  One user, who had only just joined the organisation, had seen the name O’Donnell and input O”Donnell.  The system thought that was a field separator and it threw all of the following database updates out by one field.  One of the following fields then trampled all over system memory and the machine just died.

 

We, as the test team, had tested this input field, we thought, to destruction and one of the characters that hadn’t been tried was a double quote.  The tester had used equivalence partitioning on the field and he didn’t choose double quote as an invalid input.  As a black box tester, he didn’t know that the internals of the system used double quote as a field separator.

 

Morals of this story:

 

- Test techniques won’t catch every bug. I know that this is not much of a moral, every tester already knows this, but it is sometimes worth reminding ourselves.

- Knowing something about the internals of the system can help you test more effectively

- Having at least two levels of logging (brief and detailed) in a live system can help you diagnose problems.  There were three levels on this system, level 1 was very brief, just a record of function calls, very small impact on performance, level 2 was less brief, recorded module calls, more impact on performance and level 3 was detailed showing all the user input at every screen, with even more impact on performance.

- Don’t agree to be the Support Manager.  I hated it, I much prefer Test Management.

You know, sometimes you get tripped up by being too narrow minded and believing what you see or hear is the truth.  Here is a little story that happened to me this week that I thought I would share – maybe others will learn the lesson, maybe I will next time!

 

We have had a problem in live running with one user (out of many thousands) who was complaining that on one particular screen the system kept freezing.  The user has complained a number of times and no-one could reproduce the problem.  His machine was swapped out, his mouse changed, keyboard changed all to no avail, he continued to report it was freezing. The help desk had repeatedly spoken to the guy to determine what he was doing and he seemed to be doing everything correctly.  We, in the test team tried, and failed to recreate the problem.  In the end I got one of my team to write an automated script that repeatedly went into the screen, doing something (varying combinations of input) and coming out.  We ran that script for hours and it never froze.  I told the tester to give up, it wasn’t reproducible.  Then the customer started to get letters from the user complaining that the system was unusable and, not surprisingly the customer put pressure on us.  Exasperated, I looked at all of the calls that this user had made to the helpdesk and he had made a lot of calls.  On examining each of the calls to do with freezing screens they all looked to be the same and no extra information, then on one call he mentioned a blank screen appearing.  My test expert in this area immediately said “I know what he is doing”, ran off to her test machine and recreated the problem immediately.  The problem was not that the screen was freezing, the screen was very much alive which is what had fooled us all.  The cause was that the user, instead of clicking on a hyperlink to select an item was clicking and dragging the item to another part of the screen (frames are wonderful!) and the browser had gone berserk.  Depending on what he clicked and where he dragged it to there were different results, a blank screen, a screen with hyperlinks on that wouldn’t work or the wrong screen.  My tester had raised a bug report on this over 2 years ago and it had been rejected as “will never happen in live” and the customer had agreed with that assessment and I had gone along with the decision.

 

Morals of the story. 

 

Don’t assume you know what a user means when recording a problem.

Don’t rely on a tool to recreate an intermittent problem. 

If the tool doesn’t show up the problem, don’t assume it doesn’t exist.

Bad decisions may come back to haunt you!