I turned 50 last year.  No big deal, no big party, I didn’t want one.  After all I don’t feel I am getting old or even middle aged, in many respects I still feel the same way I did when I was in my 20s.  There are signs, however, of getting a bit older.  I was reminded of these signs over the holiday weekend just gone in the UK when I was out shopping with my wife:

 

·        I look at some of the young people and think “how can they wear those clothes?”

·        I hear some of the popular music and think “but it all sounds like noise to me”

·        I reminisce with my wife and friends and we talk about things that we did when we were “young” and realise that we saw things that happened well before current teenagers were born

·        I used to think I knew it all, as I get older the more I learn the more I realise there is to learn

·        Conversations often end with the phrase “which was a lot of money in those days”

 

Maybe I am turning into my Dad, they say we do eventually.

 

I was thinking more about this today as I drove to work listening to a Jimi Hendrix CD and realised he has been dead for 36 years.  My first major row with my parents was over him, they wouldn’t let me go to see him in concert because he took banned substances and they said I was too young to go to Woodstock (they were probably right, but what a gig that was and how envious I was of the people I knew who did go!).

 

What has all this rubbish got to do with Software Testing, you may well ask!  I was thinking, whilst listening to “Voodoo Chile”, about my early days in Software Testing and there are signs of getting older:

 

·        I see a question on a forum from a newbie and think “how can anyone not know that?”

·        I hear someone talk about how they found a bug and think “but that is how we have always found that type of bug”

·        I reminisce with colleagues and we talk about how software development used to be and realise that some of the computers we used were built well before some of the current new testers were born

·        I went into Software testing thinking I knew it all, as I get older the more I learn the more I realise there is to learn

 

One of the signs of getting older is less tolerance for the less experienced, the less knowledgeable, the “young”.  It is something I see on forums that the experienced people can be terse with the inexperienced.  Maybe we “oldies” should just step back and think “That was me 25 years ago, I needed help with the basics and I didn’t know it”.  I try to be patient with inexperienced people and know that I don’t always succeed, but that is my problem, not theirs.

I know I have got fed up with answering standard questions on forums and haven’t done it for some time, maybe I should start again, with more patience, after all, they will be paying for my pension in a few years time!

 

And, lastly, I remember my first Test department annual budget which was £150k, which was a lot of money in those days.

 

 

I, and others, have written a couple of blog entries on how activities outside of software testing can help understand testing (I refer to my entries on Dancing and Gardening).  Here is a story that has happened to me over the last couple of weeks that show the reverse, i.e. where testing skills have helped in my non-work life.

 

My wife is very interested in alternative therapies, herbal remedies, Reiki, crystal healing and the like.  This in turn has made her interested in the more mystical aspects that seem to go along with these alternative therapies, such as reading and understanding Auras, Angels and so on.  Their overriding philosophy is one of openness and honesty in all dealings – remember this for later!

 

Because my wife has this interest, I have also now got a (mild) interest although I am much more sceptical than her.  It is interesting, though, how a lot of the written works in this area have a direct correlation with some of the management books I have read, particularly on managing people, motivation and intuition.

 

However, I digress.  There is a company local to us that has a shop selling things like crystals, incense sticks, books, etc.  They also run a number of courses on the types of healing and how to become more in tune with yourself.  My wife is a frequent visitor to the shop and knows the owners quite well.  

 

A couple of weeks ago the company sent out an email asking for an investor/business partner and were offering a third of the business in return for the investment as well as asking for the investor to participate in the management of the business.  My wife was interested, so we asked for information.  They told us that they wanted £x for the third of the business and that the investor could expect to make about 6% return.  I thought £x was a bit high, so after some thought, we asked the following, basic, questions:

 

·                What was the investment to be used for

·                How was the business value of £3x calculated

·                What was the percentage return on investment for the previous year.

 

I also told them that depending on their answers I would then want my accountant to get involved.

 

All went quiet for almost a week.

 

We then got an email saying that they had decided to put the business up for sale.  The price?  £x/2.  So they had valued the business for an investor at £3x, but were prepared to sell it for a sixth of that value.  It smelled very fishy.  We suspect that the £x investment was to clear debts and the business is in trouble.  So much for openness and honesty, it feels like they were trying to con us.

 

So, by asking simple questions that you would expect a tester to ask (or, indeed any potential investor) we avoided a potentially damaging financial loss.

 

The lesson from this story is that even though someone would appear to have the highest values, you still cannot fully trust them.  This can be true in Testing.  Even the most honest appearing Project Manager can try to hide things from you to get what they want.

 

Bottom line:  If it looks like a duck, walks like a duck and quacks like a duck, you still need to do a DNA test to show it really is a duck!

I like IDMTDT testing, it can be a rich source of errors and can show up data integrity faults and usability problems that you might not otherwise find.  The big advantage is that you can then see how the system handles CRUD activities in unusual combinations.

 

Oh, sorry, IDMTDT stands for “I didn’t mean to do that” and is a set of tests that try to recover from when the user has done something stupid and wants to undo somewhere down the line in the Business process. 

 

A simple example:

 

User creates a new record for a new customer

User enters name/address/Post code

User then creates new accounts record for new customer

User puts credit limit on new customer account

User enters sales representatives name

User activates new customer.

 

At this point, User finds that the Post code is mistyped.  How do you recover?  If the system allows an edit of the customer field, then it is easy, but lets say, for the sake of the example, that there is no edit function for the Post code, it maybe a primary key in the database that is uneditable.  Let us also assume that deletion of customer records is not allowed as the system requirement is to keep all records for audit purposes and the design assumes that the user does not make this type of mistake. 

 

The recovery from this type of problem in a real world situation can be a nightmare (I have seen this type of scenario in a number of systems).  The testing can, however, be quite fun as you have a number of points at which you can try to recover the situation; after the new record created, after the account set up, after the sales rep entered, after activation.  Each of these scenarios will have a different set of tests and may involve exercising different functionality.

 

Planning and scripting such testing tends to be difficult as it is often the type of scenario not covered well in requirements or business scenarios (because people don’t make mistakes, do they?), so the best technique I have found is to include it in Exploratory tests.

 

It is a type of testing that can be forgotten or is done haphazardly.  It also, as a general guideline, should be done once the software is relatively stable as you are not trying to find functionality bugs, but business flow bugs.

Introverts can’t be managers.

 

I have heard this statement from staff during objectives reviews and career counselling sessions.  I like to ask the question “Do you think I am an introvert or an extrovert?” and invariably the answer comes back that they think I am an extrovert.  Wrong! I have taken a number of personality tests over the years and I am a very strong introvert.  I look at all the attributes of an introvert and I fit them perfectly, yet I have been in managerial positions for 20 years and I like to think I am reasonably successful at it.  So why is there this impression that introverts can’t be managers?

 

I have no evidence for the following, it is merely my opinion based on my observations of people and my own self-analysis. 

 

Let us look at the main properties of an introvert:

 

The natural inclination of an introvert is to hide away in a corner on their own and get on with whatever they have to do; they keep ideas in their heads without vocalising them; they think about things quietly; they are difficult for others to know well and only let a few people in close to them.  

 

The key phrase in the above sentence is “natural inclination”.  This is not a bad thing (says the introvert!) as that is what you are most comfortable with and the way that you are most effective.  However, that is not too good for a manager, especially a test manager, who spends a lot of their day communicating with people.  

 

How do you know if you are introvert or an extrovert?  I would suggest that you take the Myers-Briggs test as I have found it to be the most reliable measure (see http://www.myersbriggs.org/) and get yourself tested.

 

So how does an introvert become an effective manager?  I can only go from my own experience, my own way of dealing with being an introvert and from the advice and guidance I have given to other introverts. 

 

The key message is to break out of your comfort zone.  Force yourself to take on attributes of an extrovert.  Note, I am not saying that you should try to change your personality to be an extrovert as that is not good for you personally, but in your professional life “act” like an extrovert.  A Myers-Briggs consultant called me a “learned extrovert” which means that I learned how to behave like an extrovert when the situation demanded it.  

 

This learned extrovert behaviour is difficult for us introverts and takes time and a lot of practice and you never stop learning.  As an example, one of the worst nightmares for an introvert is giving a presentation, especially to people you don’t know.  I had to give a presentation to 150 people, most of whom I didn’t know.  I was highly stressed before the presentation and dreaded it, but I pushed myself to perform and I got great reviews for the presentation.  I have given training courses in the past, still do the odd one, for people I have never met and this is highly stressful as you are performing for 1, 2, 3 or 4 days.  Again, you have to push yourself to do it.  It gets easier the more you do it, but it never stops being difficult.  One of the training courses I give is on how to do presentations and I inevitably get the introverts saying “but I don’t like doing it and you make it look so easy” Yes, I do, but I am quaking inside, it is a matter of managing your nerves and not giving in to them.

 

You don’t have to give presentations or training courses to “get over” being an introvert, just push yourself to speak out more when with others.  You will feel uncomfortable, that is natural for you, but force yourself.

 

So far, I have concentrated on the negative aspects of being an introverted manager, but there are positives as well.  We are generally better at managing other introverts as we understand them and can empathise.  We think things through before opening our mouths and therefore, hopefully, have a more coherent conversation than the loud mouthed extrovert who thinks with their mouth open.  This can be a useful trait when you are trying to persuade.

 

An important point.  If you are introvert, do not try to suppress your introverted nature, you can make yourself ill (seriously, it can bring on illness).  So make time for yourself, make sure you take a lunch break on your own sometimes, make sure you do have think-time.  Personally, I find that meditation is very effective for me although some people find it useless.

 

Finally, then, a message for all introverts, you can be an effective manager.  If that is what you would like to do, go for it.

Following my entry entitled “Are you positive about being negative”, Joe Strazzere asked what do I call the tests that we execute when we run out of time.  This is a good question and one that I should have answered in the original entry as Joe has correctly pointed out that I now don’t have a name for them.  

 

One of the reasons for not putting a tag on this type of testing is that it can cloud the thinking when determining what tests to run.  “Happy path”, for example, would indicate a certain type of test.

 

If you need some type of “tag”, the best I can come up with is MAT, which stands for “Most Appropriate Tests”.  Let me explain what I mean.

 

Whenever you are short of time for some tests you have to determine what tests you will run to make the best use of the time available. The thought processes go along the lines of:

 

• What are the risk areas of the system
• What are the areas that have spawned the most bugs recently
• What are the most business critical areas
• What are the areas that would be most embarrassing if they failed
• What has changed recently and therefore could have de-stabilised the system

 

So, the upshot is that I can’t give these tests a generic term, the answer is “it depends”, you should run your MAT suite!

Software Development can be a lonely business.  There is the developer sat at his/her desk churning out code day after day with only the compiler to talk to.  The only light relief that a developer has is when the tester finds a bug and needs to talk about it.  The joy of interaction with something that doesn’t just say “Syntax Error” is something to behold.  Watch the developer’s face light up at the chance of human interaction, at the opportunity to exercise vocal chords rusty through lack of use, a conversation!

 

As testers we should support these poor, lonely creatures.  Please give as much time as you can to this worthy cause.

 

Are you positive about being negative?

 

I don’t believe there is a distinction between Positive testing and Negative testing.  I don’t believe that there is such a test as a “Positive” or “Negative” test.

 

There, I have said it and the heavens haven’t opened up and I haven’t been hit by a thunderbolt.

 

You often hear the question “What is negative testing?” and you get various answers.

 

Some definitions I have read/heard:

 

1. Those test that are designed to return an error, using invalid input
2. Tests that are designed to cause the system to crash. 
3. Tests that are designed to make the system do something “off spec"

 

With reference to definition 1, so if I have an integer field and I put in an alpha character I expect to get an error, yes?  So what is negative about that test?  I have made a positive input and expect a positive response, i.e. an error message. 

 

With reference to definition 2, so, if I set up corrupt data I expect the system to crash, yes?  Again what is negative about that?  I have made a positive decision to try to force the system into an error condition to see how it handles it.  Where is the negativity?

 

With reference to definition 3, so, I am doing something to the system that might make it do something unusual.  Is this negative?  I have designed tests that try to force the system to do something it shouldn’t.  This is the closest that I can come to saying it is a “negative” test, but even here I am making a positive decision.

 

If one of the primary reasons we test is to find bugs, then we should be designing tests that are targeted at finding bugs.  Any means at the tester’s disposal should be used to find those bugs, use whatever techniques you have in your toolbox, whatever tools you have and you will execute “just tests”, not positive, not negative.

 

What does it matter?  My concern with classifying tests as positive or negative is that it is an artificial classification that clouds tester thinking.  There are no positive or negative tests, only tests that exercise the system looking for bugs.  I don’t understand the reason why testers want to distinguish between positive and negative, what benefit does it give?

 

I have heard people say “we are running out of time, only run your positive tests”.  But which are those?  I have a set of tests, I can’t differentiate them according to positive or negative.  What people usually mean by that statement is “Only run those tests that exercise the system the way the user will normally use it”.  OK, I can identify those tests, they are sometimes called “Happy path” tests, and we always create those, but they are a small percentage of the tests we prepare, I estimate less than 10% of tests fall into that category.  Are they positive tests?  By some definitions, yes, but what if one (or more) fails? Does that make it a negative test?  Again, does it matter?  A failure is a failure whatever classification is used for the test.

 

I don’t want a tester to go off and run a set of “positive” tests, I want him/her to execute tests that exercise the system in such a way that have a reasonable degree of probability of finding bugs.  That should be by exercising the system in the way the user will normally do it and doing other things as well.

 

I guess that I am not going to suddenly stop all the testers in the world from using “Positive” and “Negative” tags for tests, all I can do is try to stop my own testers!

 

Computers are more important than human life?

 

Way back in the early 1980s I worked on a large mainframe computer in London.  This beast ran at an amazing speed (for then) and generated a significant amount of heat.  The computer room was the size of a football pitch and contain latest state of the art hard disk drives, tape drives, printers, etc.

 

The computers were cooled using halon, so there were massive tanks of the stuff circulating round the hardware trying to keep it cool.  The management’s main concern was a fire as there was no offsite backup of the data and so they installed halon gas extinguishers into the ceiling of the computer room which would go off in the event of smoke being detected.  They calculated that the data was safe for about 10 seconds in the event of fire, so set the halon to go off 7 seconds after detecting the fire.  

 

For those of you who don’t know, halon has the property of removing oxygen so the fire would go out.  One tiny detail was overlooked – humans need oxygen to live.  If you were at the far end of the computer room, i.e. furthest away from the door, we calculated we could not run to the door in less than 20 seconds, even at panic speed.  Given the efficiency of halon, we reckoned we would be dead in less than 15 seconds.  But, it got worse, 10 seconds after the fire alarm went off the lead lined exit door was locked automatically to prevent any fire spreading, so even if you were half way down the room you wouldn’t get out in time.

 

Where were the system consoles positioned?  You’ve guessed it, right at the far end of the computer room.

 

Of course the management were very concerned when we pointed this out to them, they immediately sprung into action and moved the system consoles to just near the door – 2 years after we notified them of the problem.

 

RSS Feed - now on page.  Should work, at least it does for me when I put it into my reader.  That will be quite useful as now I will be notified that I have updated my blog......

 

I think I need a lie down.

Why do you like being in Testing?

 

This is a question I am asked fairly often by people who both know me and new acquaintances.  Sometimes the question is couched in the “so, you can’t program so you test?”.  As an ex-developer, I can scotch that one!  My answers are various, along these lines:

 

• I enjoy seeing the bigger picture that testing gives you.
• I like the challenge that testing gives you
• I like the variety that you get in testing
• I like finding problems
• I have a questioning mind
• I get a kick out of seeing better quality products than there would have been had I not tested it
• There is so much to learn in the field of testing
• Etc, etc.

 

If I analyse these answers, they are quite bland, they don’t really get to the heart of the reasons and they don’t really say what I feel.  Anyone who has done any consulting or psychology will tell you that there is a technique that you can use with people to “bottom out” an answer, that is, you find out the fundamental reason for why someone does something.  

 

I decided I needed to bottom out why I like testing, just for my own benefit.  After some careful thought and analysis I came up with “I like being in Testing because I like being around testers”.  In other words, it is a people thing, I get a good feeling by being around people who have a testing mindset who are thought provoking who are questioning and who make me laugh (as most testers do!).  My natural instincts are to be with these type of people and always has been, well before I got into testing.  I, like a lot of testers, fell into testing by accident and felt at home once I was there because of the people.

 

I have yet to answer the question honestly in public and say “Because I like testers”, because the questioner probably wouldn’t understand the answer.

 

 

As a Test Manager you get asked all sorts of questions and usually they start with words like “How do I …”, “When should I….” or “What should I…..”.  Occasionally, you get the “Why do I/we….”.  It is these questions that I like.  The How, When or What questions are usually fairly easy to answer, you follow your experience, process, knowledge to answer them.  The Why question is much harder and, therefore, much more interesting because it is these questions that make you think “Yes, why do we…”.  The situation on any project that you have been on for any length of time is that you start to accept the situation you are in, you have done things the same way for some time and it works and where it doesn’t work you try to change it, but you are generally in a comfort zone where everything is familiar and almost routine. 

 

A friend of mine years ago made an interesting statement.  We had been on a project for a couple of years and we were being well paid for a challenging, but relatively routine piece of work.  He said “We are in a fur-lined rut”.  That phrase made me think and ask myself “Why am I staying here?”.  Within a few months I had left the company because my friend was right, I wasn’t moving forward and I wasn’t in a position within that company to change it (long story!).  It is the “Why” question that should keep being asked and if you get the answer “I don’t know” or “Because we have always done it that way” or “If it ain’t broke don’t fix it” then it is time to get things changed and improved.  This is the essence of improvement, be it personal improvement, process improvement or any other type of improvement.  

 

The answer to the “Why” question may be that there is a good reason why and we don’t need to change, but the very asking of the question makes you go back and re-evaluate why.  If you come up with the answer that you don’t need to change, that is OK, but then you have made a conscious decision not to change for good reasons, not because you are maintaining the status quo.

 

What made me think of writing this entry was that recently one of my team asked me a “Why” question that made me do some thinking.  It was a good question and a hard one to answer.  As it happens, I am not changing anything as a result of the question, but it made me think hard and that is a good thing. 

 

Sometimes managers are not good at listening to the “Why” questions with an open mind, especially if the question comes from someone who is less experienced than them.  Sometimes team members are not very good at asking the “Why” question as they think their manager knows better than them or it makes them look stupid or it will seem that they are criticising their manager’s judgement.  

 

Managers should encourage the “Why” question and should take time to ask themselves the “Why” question.  It is often the only way to get us all to think rather than do.

 

Maybe we should have a "Why" question day on a regular basis?

Here are the tasks I have to perform in my “typical” working day:

 

• Read approximately 100 emails, most of which I have been cc’ed on or forwarded on with an “FYI” attached.  Most of which I really didn’t need the information, but you have to read it just in case there is a snippet that you really do need
• Respond to approximately 15 of the 100 I read.
• Send on approximately 15 of these emails to interested parties (with FYI on the front, naturally!)
• Write approximately a 15 emails.
• Attend, on average, 2 hours of meetings.
• Be interrupted umpteen times by all manner of people
• Analyse the previous days bug stats
• Check progress against plan
• Answer approximately 5 phone calls
• Make a number (can’t quantify) of decisions varying from the trivial to the critical
• Assist team members who need help
• Liaise with the customer at least 5 times

 

You can see that I don’t do much in the way of productive work.  I am primarily interrupt driven and I need to do parallel processing.  

 

Why am I telling you this?  Not to get sympathy, nobody gives sympathy to a manager!  I attended a talk by Scott Barber, who I have a great deal of respect for, and part of the thrust of his talk was that managers don’t listen to Performance testers and don’t understand what they are saying, i.e. we managers are not good at attending to the needs of non-functional, technical, testers.  

 

To some extent, I agree with him, we are not good at it.  I would generalise the point even further, though, in that it is not just Performance testers, it is all testers that are in our team.  

 

My rebuttal to Scott in an email I sent to him went along these lines.  I have a lot of people and things that are vying for my attention.  I prioritise my time by classifying each item requiring my attention by Importance and Urgency.  If something is high Importance and high Urgency, then it gets my immediate attention, conversely low Importance and low Urgency are unlikely to get any of my time.  I believe it is the responsibility of any of my team to assist me in establishing the Importance and Urgency of what they want from me.  In my experience, technical testers are the worst at this.  I am reasonably technically minded, but I can be (and sometimes am) blinded by the language that these guys use.  So if I am told by a Performance tester, for example, that the Pareto curve on the disk accesses on the EMC disk for physical buffering is causing them concern, I might start to glaze over.  If they, however, tell me that the Service Level Agreement will not be met if we don’t improve the performance of the database writes, then I am very interested.  

 

I am not asking for things to be dumbed down for me, I just want to have the headline as to what the problem is.  I can then delve down, if necessary, and understand what the low level reasons for the problem are and get it explained to me.  If I am given the low level information first, I can’t work out for myself the level of Importance and Urgency.

 

So, help us poor managers, give us a clue as to why we should be interested in your problem/issue/concern.  If you don’t we won’t be any the wiser, we may dismiss what you are saying when we shouldn’t and we will come across as not caring or as stupid.  We have a lot of demands on our time and I don’t mind being managed by my team and influenced as to what I should spend my next few minutes on, in fact I welcome it.  

I often try to compare activities outside of the testing world, or indeed software development, with how we behave within testing – see my blog entry on Agile Gardening.  This blog entry is in a similar (slightly whimsical) vein.  I find it useful to do the comparison as testing is such an abstract thing with nothing you can touch, smell or feel at the end of it as it is primarily a set of thought processes.  You might say that the test scripts, the screen dumps, the output files, the bug reports etc. are physical outputs of the testing process and they are, but they are not a record of the behaviours of testing.  It is this behaviour that makes us good/bad/indifferent testers.  Maybe I can explain what I mean with this comparison.

 

I like dancing, I have been Ballroom and Latin American dancing for some years now and I am slowly getting better at it.  I am told that I am one of the better dancers in my club.  I wasn’t always so.  When I started dancing classes I had two left, flat, feet, I shuffled round the floor like an arthritic robot (can robots get arthritis?).  I watched other dancers glide round the floor with apparent ease, looking graceful, masterful and in control and I wanted to be like them.  I have done my academic time (more than I really wanted to) so I know how to read books and learn from them, so I got some books on how to dance.  I read the books and I still couldn’t dance any better.  It took me some time to realise that the only way I could dance better was for me to have a teacher who showed me the basics, who showed me how to do well the simple things, who encouraged me and told me when I got something wrong (which was a very frequent occurrence!).  In short, I had to do, not read.  Now, the books make some sense, I can learn some variations and techniques from them but I then have to practice them repeatedly with someone who already knows them to become good at some of the moves.

 

There is no tangible output from being able to dance, but you know someone is good when you see them.

 

Does this ring any bells as a parallel with testing?  Of course it does – I won’t spell it out here because if you can’t see it, you need to go do some testing and stop reading my blog!

The project is being outsourced, hey-ho, never mind, not sure it is a good idea, in fact I am sure it isn’t but that is what is going to happen and nothing I do or say is going to change that.  But, this blog entry isn’t about outsourcing and the pluses and minuses, this entry is about recruiting your successor.  

It is a strange experience trying to recruit someone who will be doing your job and when they do you will be out of a job.  Now, now, put those hankies away, I have been expecting this for over a year, I am a contract test manager so expect to have periods of unemployment, it goes with the territory.

I have been trying to recruit my replacement for some months now and it is proving to be extremely difficult.  Here are the difficulties I have had:

·          The quality of test managers in the favoured country for outsourcing has been incredibly poor.  

·          The problem I have with rejecting people is that it could be seen as me finding fault so that I keep my job.  This is simply not true as I am a professional and will behave professionally at all times, regardless of personal thoughts.  Besides, I know I know am leaving so I want to have the best person I can find to take over from me.  I have made sure that for every interview I have had a senior permanent member of staff in the interview who can verify that I have been fair in my questions and answers and that my decision is a purely professional one. 

·          It is very difficult to interview someone over the phone and that being the only interview you can have and then you have to make a decision based on the telephone interview, but that is the process.

·          Despite being professional, see second point, I have a real affection for this project and the test team, most of whom I recruited and I want someone who has the drive, intelligence, testing skills and people skills that will enable the team to progress and not regress.  I have to ask myself “Am I too picky?”  I don’t think so, but I keep asking myself that question.

So why are the test managers in the favoured country so poor?  Obviously, I have not interviewed EVERY test manager in the country, so maybe my sample was just bad.  Here are some of the problems I encountered with them:

·          Poor grasp of English.  As the customer is English and does not speak favoured country language, English has to be good, not just adequate.  This role requires considerable customer facing skills.

·          Poor grasp of testing.  I expect a test manager to at least understand the concepts of testing and I want someone who has a good understanding of testing.  Not everyone agrees with me, some people say if you can manage one thing, then you can manage anything.  There is something in that, but on this project the test manager NEEDS to know a lot about testing.

·          Poor people skills.  Maybe it’s the culture of favoured country, although having managed a number of people from the country for some time I don’t think so.  I expect any manager to have decent people skills, it is part of the job.

·          A lack of understanding of what the test management role is.  These people are being put forward to me as experienced test managers and they do not understand some of the basics of what a test manager does, e.g. managing bug statistics.  I find that incredible.

So, recruiting your successor is not easy, I have found one good candidate so far out of many put forward, hopefully he will turn out OK.

And to sign off, this is the priceless statement one interviewee made “System Testing is a dumb task, you need no skill for it.  The real skills are in User Acceptance testing.”  My fellow interviewer said afterwards “Fortunate it was a telephone interview, I think you would have killed him had it been face-to-face”. 

Oh, and just on the of-chance that anyone was wondering where I have been for the last 3 months since my last blog entry, the project has been taking all of my time.  I am still up to my neck in muck and bullets, but I will try to keep on blogging when I get a spare few minutes

Well, philk10 has been nagging me to write up last week’s SIGIST, so here goes.

 

Just for those who don’t know what this is, it is the British Computer Society Special Interest Group In Software Testing (heck of a mouthful, so SIGIST will do).  A one day conference is held every quarter and the quality of the speakers has improved considerably over the last few years, including some world known people.  Last week’s was well attended, especially considering there was a very important football match on that day.

 

The SIGIST this time had Johanna Rothman as the guest speaker.  She did three sessions, so she must have been really tired by the end of the day!

 

First session was entitled “Are you Hiring Yesterday’s Testers?”.  The presentation focussed on what a first class tester is and how to recognise the difference between first class and second class testers.  This is not concerned so much with the ability and skill of the individual tester, but attitude, maturity and skill of the hirer.  The following list, taken directly from the conference handouts, shows what first class testers looks like:

 

• Has a wide variety of testing skills
• Finds the most atrocious problems before release
• Developers enjoy working with the testers
• Learn about risks and other product information
• Can quantify the risk of a release
• Accurately predict necessary testing time and can explain why
• Understand product release is a business decision
• Supply enough information to predict rework and help projects complete on time

 

The following list, again taken from the handout, shows what second class testers look like:

 

• Routinely excluded from requirements or design meetings
• Resort to eavesdropping to hear information about the product
• Request for tools postponed or ignored
• Training budget significantly less that developer’s budget
• Interchangeable, i.e. they have such similar skills it doesn’t matter who you assign to which products
• Only work with developers on code artifacts because they either aren’t brought into the project early or they don’t know enough about the early phases to supply feedback
• Work all hours because the developers have so many defects to fix

 

Do you recognise any of those attributes from either list in your group?

 

Johanna went on to discuss how to create a first class group, which boiled down to making sure that the first class list above was achieved.

 

She also talked a bit about how to hire first class testers and it was a very short précis of her book “Hiring the Best Knowledge Workers, Techies and Nerds:  The Secrets and Science of Hiring Technical people”.

 

It was an entertaining presentation that I saw a number of people nodding their heads in agreement (no, they weren’t falling asleep!)

 

Johanna’s second session was a workshop entitled “How to Become a Sought-After Tester or Test Manager”.  This was a workshop designed to make you evaluate your skills and determine an action plan to improve them.  I think it would have worked well with a small group, but the group was large and it didn’t have the dynamics that a small workshop would have had.  Interesting, but not as useful as I had expected.

 

Johanna had the last session of the day entitled “Successful Software Management: 15 Lessons Learned”.  This was a superb presentation and I can’t do it justice here.  Basically, Johanna was going through what makes you successful as a Technical Manager.  The 15 lessons were:

 

1. Know what they pay you to do
2. Plan the work
3. Accept only on Number 1 priority at a time
4. Commit to projects after asking your staff
5. Hire the best people for the job
6. Preserve good teams
7. Avoid micromanaging or inflicting help
8. Treat people individually and with respect
9. Meet weekly with each person
10. Plan training time in the workweek
11. Fire people who can’t do the work
12. Emphasise results, not time
13. Admit your mistakes
14. Recognise and reward good work
15. Manage yourself

 

The presentation was peppered with her own experiences, mistakes and successes and whilst most managers know all these things, how many of us actually do ALL of them?

 

I went to two other presentations, the first was by Isabel Evans, entitled “A Balanced Scorecard Approach for Assessing Test Value and Success”.  This was interesting as I had heard of and seen the Balanced Scorecard Approach, but what Isabel has done is to adapt the approach for testing and how we can apply our metrics to show our value and success to the organisation.  I could see how it could be useful and I could also see that some of it might take a lot of time to put together, so I still have to think through how I can use what Isabel said in my own shop.  Her message is that what we report must be of use to the reader in their own language.  An example:  We are good at producing defect statistics, X raised this week, Y still open, Z closed, but what does that mean?  We should be couching our report in language that the reader is interested in, e.g. “the defect rate is such that the release will not be stable by next week”, or “the cost of rework due to the defect level is £x.  Her basic conclusions were as follows:

 

• Any organisation needs a balance of financial, customer, internal and innovation measures
• Test Teams exist to provide information
• Test Teams’ customers include the organisation, the project, the managers, the builders, the supporters, the users and customers and other measurement groups
• Our measurement and reporting help us
• To help our customers our measurement and reports need to reflect their goals, targets, indicators and concerns

 

The second presentation was by Graham Thomas, entitled “Seven key measures for Software Testing”.  This presentation took us through a horrendous example of a weekly test report that he had found, he showed the problems with it and then presented seven measures that help the weekly report become useful.  To be honest, there was nothing in the presentation I didn’t know and don’t use in one form or another, but Graham was a good speaker and made it entertaining.  His main focus was on showing test progress, script progress, risk profile, risk mitigation, Fault S-curve, Environment availability and Coverage.

 

The were other sessions I didn’t go to, because they clashed with the ones I did go to.

 

Conclusion: A good day, I learnt a lot.  This SIGIST was very test management biased whereas other SIGISTs have been very testing biased, so it made a nice change.  Oh, and England did win the football match…

A sledge hammer to crack a nut?

 

We are about to start testing a major enhancement to the system.  Development will be delivering this enhancement in two drops, firstly the back end and a month later the client end.  This has meant that we have had to write a utility to create the packets that the client would have sent to the host in order to test it.  This Perl utility gets responses from the tester, formats the responses and saves them away to a file for processing on the host.

 

My “plan” was to then use QuickTest Pro to be able to create many of these files and files with many packets, using a spreadsheet to drive the QTP script.  I estimated that it would take about a week to get this working as I wanted it.  Unfortunately all my QTP resources were busy on other things and could not get to it.  Panic was starting to set in until I was in the shower this morning and had a “Eureka” moment – or more likely I realised I was being stupid!  I could do the self same thing with a small VB script in Excel.  When I got into work, I spent hour and a half putting the script together, testing it and hey presto everything I needed.

 

I know a lot of people state home grown tools are often the best, but I wonder how often we plan to use the tool we have without really thinking about the best way of doing it?  I saved my team a week’s work and got something for free – well, I am a Manager, so my time doesn’t count J.

Developers – don’t you just love ‘em?

 

As part of releasing a new version of the software to Live, we perform a sanity test on the release day to check the release has gone Live OK and/or to be aware of potential problems when the users start using it after the release.  One of my testers found a problem with the release during this sanity check that was not evident in the test environment.

 

Conversations went something like this:

 

Tester:  I have found a problem with our Master/Slave installation

 

Me: Well, raise a bug report, high priority

 

Some few minutes pass as Tester tells Developer a bug report will be raised……..

 

The sound of heavy footsteps coming to my desk….

 

Developer (clearly upset):  Why do you want a bug report raised on this, it is not a bug!

 

Me:  Is there a problem with the installation script?

 

Developer: Yes

 

Me: Then we should raise a bug report

 

Developer:  But it is a very rare occurrence.

 

Me:  But not impossible?

 

Developer:  No, but it would be very unusual

 

Me:  So, it is possible and is a problem, therefore we raise a bug report.

 

Developer:  Well it isn’t a high priority problem.

 

Me:  What is the workaround then?  (Note:  High priority means there is no acceptable workaround)

 

Developer (after thinking for a few seconds):  We would send an engineer to the site with a memory stick

 

Me:  And would that be done within our SLA for this type of problem?

 

Developer:  It would only take a few minutes to do the installation

 

Me:  Even if the engineer had to drive 300 miles to the site?

 

Developer:  Well, I suppose not

 

Me:  So it is high priority problem

 

Developer:  It won’t get fixed.

 

Me:  That is not your call, nor mine – that is a decision made at the bug review meeting that I chair.

 

Developer:  To fix it we would have to redesign the system, do you want to redesign the system?

 

Me: I don’t do the design, I report the problems with the design.

 

Developer (now getting desparate):  Well, you won’t be popular for raising this.

 

Me:  I didn’t go into Test Management to be popular.

 

Developer storms off to get a doll in my image so he can stick pins in it (I suspect).

 

Some more time passes……

 

An email comes out that some Master/Slave devices in Live have had an installation problem…

 

The one statement that almost had me in fits of laughter was the “you won’t be popular for raising this”.  Did he really think that every time we find a problem that I sit down and determine whether to raise it or not depending on how it would affect my popularity? Sheesh.

Selling Exploratory Testing

 

One of the difficulties I expected to have when introducing Exploratory Testing was having to sell Exploratory Testing to the customer and to my own management.  What I didn’t expect was having to sell it to my own testers.  Some of them had used ET before, some had not.  Those that had needed no persuading, they already knew the benefits that ET gave.  Some of those that had not used it before were very sceptical and could not see how you could test something without scripting it first.  

 

This surprised me, as testers, by their very nature, explore the system under test whether they are scripting or not (see James Bach’s articles on this on www.satisfice.com).  I tried to determine why there was this resistance to a way of testing.  The answers I came up with are, inevitably, based on a very small sample of testers (c40) and are, therefore, just my view on what I saw.  [Perhaps someone will want to pay me for a larger study J]

 

1) The way a lot of us have been trained in IT has been to document first, then do.  Just “doing” is considered to be unprofessional and cavalier, akin to hacking.  So just testing can have an uncomfortable feel to someone trained that way.  

 

2) There is the attitude that if you can’t see what I have done, then you can’t measure how good I am.  This attitude stems from the CYA attitude that comes from bitter experience that if something goes wrong, you had better have some evidence to show that you did your job.

 

3) In my own shop, we had the customer crawling all over us in the early days, distrust was rife, so everything had to be specified minutely and thoroughly reviewed by the customer.  It was drummed into the test team that documentation was everything.  This legacy meant that changing the way of working was difficult purely because it was a change and some people don’t like change.

 

4) Related to point 3, there is a fear of the unknown.  Testers who had never done ET before were nervous of failing when doing something new.  Even though they were experienced testers, it was an approach that they had to learn.

 

How did I get round this resistance?  

 

Myself and one of my Test Team Leaders who was experience in ET did a number of courses to the teams on what ET is, what the benefits are and how we were going to implement it on our project.  As part of the course, the team had to do some ET.  This ET was on a piece of the system that everyone knew, as it is core to the system and supposedly very stable.  Each of the courses found problems in this part of the system that we didn’t know were there.  None of the problems were serious, none would have caused us to stop shipping, but it the made the point that ET can find problems that had not been found.

 

We also put in place Session Based Testing (as defined by Jonathan Bach) and this helped, because the testers could see that there is documentation associated with ET and that the documentation was useful, even if it is totally different than what they were used to.

 

Now ET is a major weapon in our armoury and you wonder how we ever got along without doing it in the past.

In my second lessons learned, I mentioned there was a customer “problem” on a previously working system.  Here is the story.

 

The system had been sized to cope with X number of input forms per day and performance and load testing showed that the system could more than cope with twice that number, so we were confident that there would be no performance problems.  After a few weeks of live running, the customer complained that the throughput of forms was below what was expected and we needed to improve the performance of the system.  I was surprised as the system performance monitors showed that the system was never busy and never ran out of any resources.  I checked the logs and all looked fine, there was plenty of spare capacity.  I checked the networks and they were running smoothly with no bottlenecks.  I didn’t know what to do next.

 

The only conclusion I could come to was that there was something in the way the customer was using the system that was reducing the throughput of forms, so I went to the data input room and watched them input data for a couple of hours at the busiest time of the day.  All seemed fine, the input clerks were going as fast as they could and they were keeping up with the system and the system was keeping up with them.

 

What next?  I then went to the machines that were used for analysing the data on the forms, again at the busiest time of the day and all the operators there were going flat out, but again they were keeping up with the system and the system was keeping up with them.  I completely foxed, everything seemed fine everywhere, so why was the throughput down?

 

As a last resort, I decided to follow the forms from delivery to the building right through to final analysis.  This is what I found.

 

The forms arrived in a van on the ground floor at the Goods-in area.  The forms were off loaded onto trolleys and sent in the lift to the 12^th floor.  There, the forms were taken off the trolleys, each one was booked in to a log, loaded back onto trolleys and the trolleys were then sent down a long corridor to the data input room where they were off loaded into piles for the data input clerks to process. 

 

During the busy time of the day, the trolleys were working full blast, the forms were sent to the data input room as fast as they arrived.  During the non-busy time, however, the porters would wait for a full trolley before sending it to the data input room.  This could cause a delay of half an hour, an hour or even longer between trolley loads.  The data input people were happy because to gave them a break.

 

All this meant that the throughput was delayed at certain times of the day and everything was idle.  The customer had complained that the average throughput was too low, however it wasn’t, the throughput was below average at certain times of the day.  

 

Lessons learned:

·         The customer may not tell you the whole story

·         The problem may be with the customer’s processes, not with the system

·         Don’t always assume that the busy part of the day is the cause of the problem (yes, I know that is counter-intuituve)

 

How do you know when you are “getting it right” with your customer?

 

I have dealt with customers as both Test Manager and Project Manager for some years now and I used to worry about whether I was “getting it right”.  I came to the conclusion some time ago that if you aren’t getting complaints and your management are not getting complaints about how you work with them, you must be doing alright.  Here is a little story about something that happened this week that made me feel really good about my relationship with the current customer.

 

Background:

 

I started on this project 3.5 years ago and the Test Group relationship with the customer was one of the worst I have ever come across.  They didn’t trust us, accused us of lying, hiding facts and the customer meetings I had were extremely difficult.  I had to work really hard to improve that relationship, which I think I managed to do.  One of the things I did which helped was to instigate a weekly bug review with the customer.  At that bug review, we review all the bugs that have been raised in the week (from System Test and Live) and all the bugs that have changed to a terminal state (i.e. closed or set to “not a fault”).  The purpose of this meeting was to make sure we all agreed whether it is a bug, that the severity is correct and the schedule for fixing is reasonable.  This meeting has been running for just under 3 years and has been very successful in improving openness and mutual understanding.  

 

One of the things that happens to us in software development is that a fix or enhancement can break something else or that a bug re-appears for whatever reason.  The UAT Manager at these meetings christened such bugs as coming from the “Change Fairy”.  A number of bugs over the years have been classified as having occurred due to the Change Fairy.

 

The story:

 

This week we held a “Celebration of Success” dinner which was a dinner for a number of the project team, customer and suppliers, about 100 people in all.  It was a posh do, dinner jackets, very formal.  After we all had eaten, there were presentations to a few people and speeches.  The UAT Manager presented to me a “Change Fairy”.  This was something he had made himself out of bits he had got from an old clock, a fairy that he had put on some wings, put into a cage and put on a plaque with the motto “Change Fairy – Do not Release”.

 

I think this shows how much our relationship has improved and is now very good, that a customer who had totally distrusted us can now pull a comical stunt like that at a public function.  I was struck dumb and so pleased. 

 

Yes, I think I am “getting it right” with this customer.