Sometimes I just want to scream...
Had a long meeting with a test manager last week. He has spent a lot of time on tools with his team, invested a lot of effort on making his automated test suite work for him and from what I could see done a pretty good job. He had a number of performance tests done which, again, looked pretty good, good use of the tool, good results coming back.
He then spoilt it for me. His test case management tool work was equally extensive and he had taken it to the level where every test case was in the tool, the test case execution was done through the tool and the tester had to execute every step which was at the lowest level possible (move mouse to field, enter ) and then record whether actual result met expected result, for every step. He was proud of the fact that his tool gave him all of his traceability and coverage metrics because of this.
For reasons I can't go into, I was unable to say "do you realise how dreadful this testing is?", my tongue was bleeding from having to bite so hard on it.
The effectiveness of his testing was based on metrics coming from his tool. I have never seen this to be true. The effectiveness of testing has little or no correlation to how well the test management tool works - why can't people see this?
Problem with hiring testers Pt2
We are in a recession, the unemployment rate is increasing, so why, oh why, am I finding that it is so difficult to find a good tester to hire?
I have some theories as to why there are so few people applying for my vacancy, but no real facts.
And the CVs I have received are, quite frankly, dreadful and those that aren't dreadful, when I interview it is clear that the CV lied. An example, one CV said that the person was experienced in SQL and in the interview the person could not describe how to do a simple SELECT statement! OK, SQL experience is not vital to be a good tester, but if you say you know some SQL, then please, please, be able to describe what you use SQL for and how to do simple operations.
Do applicants really think that the person interviewing them knows less than them and that they won't be caught out? I guess they must and it makes me wonder whether applicants do get through some interviews and start work doing a poor job? Probably yes, especially as these people are in work now.
It all got me thinking about the state of the testing world (again, I sometimes get thinking this way!) I am surprised at the lack of skill in testers who have 3,4,5 years “experience” and yet there are a lot of testers in the software world. I am baffled. Does this mean that there are a number of companies who have test teams made up of poorly skilled people? Does this mean that there a number of companies who are paying people relatively high salaries to do a bad job?
Now, maybe I am lucky, the company I work for values testing as an integral part of the development of software and is keen to employ good testers and every company I have ever worked for in a testing capacity has had the same view, so maybe I just haven't worked for a company where this isn't so and there are really test teams out there who are useless? Thinking about it, I did go for a job once with a company that clearly had no clue about testing – they wanted to hire me as a Test Manager and the IT Director interviewed me (very badly, as I recall) and it was clear that I would be a bit part player in the decision making process. I turned the job down. Maybe I should have taken it just to see how I could have changed things.
I know I have been accused of being picky in the past when hiring testers, and I would agree, I am picky – because I want someone who is skilled in the craft of testing or someone who has the potential to be skilled. Surely that should not be too much to ask?
This week I have, finally, filled the vacancy. Assuming he accepts the offer. It has taken me about 6 months to find him.
The problem with hiring testers
Tony Bruce asked a question on Linkedin that I just had to blog about.
Tony was asking why job adverts are so specific in their requirements for testers instead of asking for a good tester.
I think this is a good question and one that I have been struggling with for years. Let me give an example of what has been happening to me recently when trying to recruit, so from the other side of how Tony was looking at it.
What I have been looking to recruit is someone who can test. Full stop, that is what I want. I don't care if they have a background in banking, health care, games, whatever. I don't care if they have used QTP, Selenium, Robot or any other tool. I don't care if they have been using Agile, Waterfall or any other methodology. I don't care if they have Unix, Windows, mainframe or any other knowledge. I WANT SOMEONE WHO CAN TEST!!
So, I go to HR and say that. I get blank looks at first, then I explain a bit further and the conversation goes something like this:
“Oh, you want a junior tester, then?”
“No, I want an experienced tester who knows how to test.”
“So what technologies do they need to have?”
“It doesn't matter, I want someone who can test”
“What tools do they need to have used?”
“It doesn't matter, I want someone who can test”
A pause ensues.......
“OK, so what qualities do you need?”
(Methinks: Ah, we are getting somewhere!)
So I explain that I want someone who has the ability to think creatively, be able to apply testing techniques to a testing problem, be a good bug finder, etc, etc.
“Oh, you want a junior tester, then?”
I know what the real problem is with this conversation, it is that HR don't understand what testing is all about. They understand the need for technologies, tools, etc, that classifies the “type of person”. They can't understand the qualities required of a tester – or, I can't explain it well enough.
End result is that HR put the job spec that I have put together into their infernal machine that determines what the salary should be for this new hire and it comes out ridiculously low for an experienced tester, in fact it is at a junior tester level.
So, what do I do? I put together another job spec that has the requirements stated in terms of tools, technologies, methodologies, etc., re-submit that and lo and behold I get a salary level that looks about right.
The end result is that I have an advert out that states things I don't need as the only way I can get a decent salary level for the person.
That then perpetuates into the job agencies who filter out anyone who doesn't fit my “requirements” and, yes, I know I am potentially missing getting in someone who would be a good tester.
And then, when I interview people I am sure they wonder why I am not very interested in their skills that I ask for in the job advert.
Wish I knew how to resolve this problem....
One of the things I try to impress upon my testers is that when testing it is important not to “prove” that the system works correctly, but to find the weaknesses, faults, bugs. If a tester tests looking for the system to do what it is supposed to do, then that is what they will find – nothing (or very little), whereas if they are testing to find faults they are more likely to find them.
It is too easy to think “that is what the specification says, therefore, that is what I will test against”.
I fell into that trap this week. We have a lot of faults that have come back from Development and we have a short timeframe to re-test them. All of the faults are allocated out to team members to retest. One of my testers needed to complete some other work before starting on the re-tests and so I agreed that work needed finishing first, then I said “And then you can close off those faults”. After a short pause, he said “Don’t you mean that I should find the faults in the fixes?”. It was said in a jokey sort of way, but it was a good point!
Two things hit me. First, of course that is what I should have said and it was embarrassing that I had fallen into the trap. Secondly, I was very pleased that the tester had pulled me up on what I said, it means I am getting the mindset of the testers better.
Isn’t it odd though how even seasoned professionals like me can make that type of mistake?
King Midas' ears - a rant
You know sometimes you just need to rant. I was reminded of King Midas’ barber and Midas’ donkey ears today
“Once Pan had the audacity to compare his music with that of Apollo, and to challenge Apollo, the god of the lyre, to a trial of skill (also see Marsyas). Tmolus, the mountain-god, was chosen as umpire. Pan blew on his pipes, and with his rustic melody gave great satisfaction to himself and his faithful follower, Midas, who happened to be present.
Then Apollo struck the strings of his lyre. Tmolus at once awarded the victory to Apollo, and all but Midas agreed with the judgment. He dissented, and questioned the justice of the award.
Apollo would not suffer such a depraved pair of ears any longer, and caused them to become the ears of a donkey. The myth is illustrated by two paintings "Apollo and Marsyas" by Palma il Giovane (1544-1628), one depicting the scene before, and one after the punishment.
Midas was mortified at this mishap. He attempted to hide his misfortune with an ample turban or headdress. But his hairdresser of course knew the secret. He was told not to mention it. He could not keep the secret; so he went out into the meadow, dug a hole in the ground, whispered the story into it, and covered the hole up. A thick bed of reeds sprang up in the meadow, and began whispering the story and saying "King Midas has a donkey's ears."
Today I feel like King Midas’ barber – I can’t keep this in any longer.
Something that really makes me cross is when so-called industry experts believe that they know better than you how to solve your problems. You see it in response to questions in forums, you see it in people’s blogs, you hear it at conferences and you hear it in conversations.
An example: I was at a conference and in the evening got into a conversation with one of the speakers. This speaker (who I will not name) is well known in the testing world, has written articles and books, teaches and speaks and is well respected. We talked a bit about generalities of testing and then got talking about what I was doing and his only retort was “I’ll give you my card, I would sort it out”. How could he possibly know he could sort it out? Why did he think he could do a better job than I could? He didn’t know what the context was, the constraints or anything about the project. The arrogance just stunned me and I couldn’t think of a suitable response.
My particular problems are split into a number of categories:
1) Those that are imposed as constraints on me from higher up. Things like the project’s budget, company policy. I can’t solve those, I have to live with them and make the best of them.
2) Those that are caused by the project methodology. The project uses a Prince 2-like methodology, which means that there are some things you just have to do. Whether you think Prince 2 is good, bad or indifferent is irrelevant, as with any methodology, problems are caused by it. Some I can work around, some I can’t.
3) Those that are caused by people working on the project but not in the test team. They have their own priorities, ways of working, constraints, etc. Some of those may cause me problems. Some I can resolve, some I can’t.
4) Those that are caused by people within the test team. These are totally within my control and I can work towards resolving them (e.g. skills gaps).
5) Those that are caused by me. Much as it pains me to admit it, I am not perfect and I cause some of my own problems. I try to work towards self-improvement, but perfection is probably about 300 years away.
I am sure that I am not the only one with this set of problems. I am glad there are problems; otherwise I would be out of a job!
I don’t believe that any one person can solve all of these problems for me. I most definitely believe that anyone could come into the project with no knowledge of its history, culture, methods and sort out my problems. If I had a free hand, had a lot of time and a lot of money I might, just might, be able to solve some of the problems this project has, but that is not going to happen.
I ask for help, advice, like most people. I am not afraid to listen to people who have tried things and found them to work. I have tried some of these methods, some have worked, some haven’t, but no-one, I repeat, no-one, could tell me how to sort out the problems I have.
So my plea to the experts is “Don’t tell me you would know how to solve a problem, give me pointers to how I can solve the problem”
Finally, here is a counter-example to the one I gave above.
I spoke to Johanna Rothman at a conference and asked her a question which was related to the talk she had just given. She did not give me the answer, nor tell me how to solve it. What she did was ask me questions about the problem and made a couple of suggestions of how I might approach solving the problem. It was wonderful, I was able to take that away and work on solving the problem myself.
Rant over. The reeds have finished their story and can now die.
My time is precious - help me use it wisely
Here are the tasks I have to perform in my “typical” working day:
- Read approximately 100 emails, most of which I have been cc’ed on or forwarded on with an “FYI” attached. Most of which I really didn’t need the information, but you have to read it just in case there is a snippet that you really do need
- Respond to approximately 15 of the 100 I read.
- Send on approximately 15 of these emails to interested parties (with FYI on the front, naturally!)
- Write approximately a 15 emails.
- Attend, on average, 2 hours of meetings.
- Be interrupted umpteen times by all manner of people
- Analyse the previous days bug stats
- Check progress against plan
- Answer approximately 5 phone calls
- Make a number (can’t quantify) of decisions varying from the trivial to the critical
- Assist team members who need help
- Liaise with the customer at least 5 times
You can see that I don’t do much in the way of productive work. I am primarily interrupt driven and I need to do parallel processing.
Why am I telling you this? Not to get sympathy, nobody gives sympathy to a manager! I attended a talk by Scott Barber, who I have a great deal of respect for, and part of the thrust of his talk was that managers don’t listen to Performance testers and don’t understand what they are saying, i.e. we managers are not good at attending to the needs of non-functional, technical, testers.
To some extent, I agree with him, we are not good at it. I would generalise the point even further, though, in that it is not just Performance testers, it is all testers that are in our team.
My rebuttal to Scott in an email I sent to him went along these lines. I have a lot of people and things that are vying for my attention. I prioritise my time by classifying each item requiring my attention by Importance and Urgency. If something is high Importance and high Urgency, then it gets my immediate attention, conversely low Importance and low Urgency are unlikely to get any of my time. I believe it is the responsibility of any of my team to assist me in establishing the Importance and Urgency of what they want from me. In my experience, technical testers are the worst at this. I am reasonably technically minded, but I can be (and sometimes am) blinded by the language that these guys use. So if I am told by a Performance tester, for example, that the Pareto curve on the disk accesses on the EMC disk for physical buffering is causing them concern, I might start to glaze over. If they, however, tell me that the Service Level Agreement will not be met if we don’t improve the performance of the database writes, then I am very interested.
I am not asking for things to be dumbed down for me, I just want to have the headline as to what the problem is. I can then delve down, if necessary, and understand what the low level reasons for the problem are and get it explained to me. If I am given the low level information first, I can’t work out for myself the level of Importance and Urgency.
So, help us poor managers, give us a clue as to why we should be interested in your problem/issue/concern. If you don’t we won’t be any the wiser, we may dismiss what you are saying when we shouldn’t and we will come across as not caring or as stupid. We have a lot of demands on our time and I don’t mind being managed by my team and influenced as to what I should spend my next few minutes on, in fact I welcome it.