SCRUM MayHem
Where have you been, Mr Nairn?
I noticed with shock that it is over 2 years since I last posted a blog entry. Why?
It is interesting that I have not stopped writing, just stopped posting here. I looked at what I have written and each of the potential entries are:-
- Not complete, either in thinking, writing or both
- Internal to the company
- Dreadful!
Partly my enthusiasm for posting stuff sort of went away due to a number of factors. In recent months I have re-discovered my enthusiasm and have started using my twitter account. I think I will start blogging again.
So, I will make this a short entry and hopefully get posting again more regularly.
One of the things that has got me more enthused recently (and I will probably blog on why my enthusiasm waned) was that I have been seeing more and more enthusiasm within my own test team in my new company and that has been a long hard slog.
I have been having some very interesting testing discussions with one tester in particular that has made me think hard and I always like to do that. Some of the results of those discussions will make great blog entries!
It is only a one line fix
We had a problem report outstanding for a long time, simple little problem. When a user gets a message in their inbox, a yellow marker shows up on their screen to indicate they have a message waiting. Some messages are important messages that the user has to action and then they get a red marker on their screen. Some users get the important messages as a FYI, but were still getting the red marker when they can't action the message and in this circumstance they should receive neither a red or yellow marker. Low severity and low priority bug.
Fix comes in from Development. The fix is to remove the marker for these users, which is a simple one line fix.
We test it. It is a low severity/low priority bug amongst all the other bug fixes we have to retest, to do a lot of testing in this area requires a lot of data set-up, so we just do a quick check. All is well.
The release goes to Live.
Everyone reading this knows what is going to happen next, otherwise there would be no point to this post! Yes, something was broken. True the users who were supposed not get their marker didn't get their marker, however, no other users were getting any marker either and this marker is relied on for important messages, as the users need to take action - no action means the user is prevented from using the system. Not getting the marker meant some users were not actioning and getting locked out of the system.
Big “oops”.
OK, so all test groups make this type of error sooner or later and we learn from them (sort of, sometimes we don't). But the whole episode made me think about Michael Bolton's distinction between “checking” and “testing”. What we tend to do on low severity bug retests are “checks”. Heck, we have already tested this area once, lets just check the fix. We might do some regression testing, maybe through running an automated suite, but we rarely test a bug fix that is low severity.. Maybe other test groups do more testing on this type of bug fix, but mostly my group doesn't.
So, I asked myself, SHOULD we have done more testing? In order to answer that, I looked back at all the low severity bugs we had found, over 3000 of them. I looked at how many of these fixes had either failed retest and/or caused a problem in Live. Few had failed retest, less than 1 percent. I could only find one other low severity problem that had caused a problem in Live and that was also low severity. I also estimated what would it have taken to test these fixes rather than check them? A lot was the answer. The rate of return for putting testing effort into these bug fixes would have been incredibly low.
So, my conclusion was that, yes it was bad that the bug got through, but I don't want to start testing low severity bugs, it will cost too much effort that would be better targeted at more important areas finding more important bugs. I will just have to take the risk that another low severity bug fix will cause a problem in Live.
Testing is all about assessing risks and acting accordingly.
Problem with hiring testers Pt2
We are in a recession, the unemployment rate is increasing, so why, oh why, am I finding that it is so difficult to find a good tester to hire?
I have some theories as to why there are so few people applying for my vacancy, but no real facts.
And the CVs I have received are, quite frankly, dreadful and those that aren't dreadful, when I interview it is clear that the CV lied. An example, one CV said that the person was experienced in SQL and in the interview the person could not describe how to do a simple SELECT statement! OK, SQL experience is not vital to be a good tester, but if you say you know some SQL, then please, please, be able to describe what you use SQL for and how to do simple operations.
Do applicants really think that the person interviewing them knows less than them and that they won't be caught out? I guess they must and it makes me wonder whether applicants do get through some interviews and start work doing a poor job? Probably yes, especially as these people are in work now.
It all got me thinking about the state of the testing world (again, I sometimes get thinking this way!) I am surprised at the lack of skill in testers who have 3,4,5 years “experience” and yet there are a lot of testers in the software world. I am baffled. Does this mean that there are a number of companies who have test teams made up of poorly skilled people? Does this mean that there a number of companies who are paying people relatively high salaries to do a bad job?
Now, maybe I am lucky, the company I work for values testing as an integral part of the development of software and is keen to employ good testers and every company I have ever worked for in a testing capacity has had the same view, so maybe I just haven't worked for a company where this isn't so and there are really test teams out there who are useless? Thinking about it, I did go for a job once with a company that clearly had no clue about testing – they wanted to hire me as a Test Manager and the IT Director interviewed me (very badly, as I recall) and it was clear that I would be a bit part player in the decision making process. I turned the job down. Maybe I should have taken it just to see how I could have changed things.
I know I have been accused of being picky in the past when hiring testers, and I would agree, I am picky – because I want someone who is skilled in the craft of testing or someone who has the potential to be skilled. Surely that should not be too much to ask?
This week I have, finally, filled the vacancy. Assuming he accepts the offer. It has taken me about 6 months to find him.
The problem with hiring testers
Tony Bruce asked a question on Linkedin that I just had to blog about.
Tony was asking why job adverts are so specific in their requirements for testers instead of asking for a good tester.
I think this is a good question and one that I have been struggling with for years. Let me give an example of what has been happening to me recently when trying to recruit, so from the other side of how Tony was looking at it.
What I have been looking to recruit is someone who can test. Full stop, that is what I want. I don't care if they have a background in banking, health care, games, whatever. I don't care if they have used QTP, Selenium, Robot or any other tool. I don't care if they have been using Agile, Waterfall or any other methodology. I don't care if they have Unix, Windows, mainframe or any other knowledge. I WANT SOMEONE WHO CAN TEST!!
So, I go to HR and say that. I get blank looks at first, then I explain a bit further and the conversation goes something like this:
“Oh, you want a junior tester, then?”
“No, I want an experienced tester who knows how to test.”
“So what technologies do they need to have?”
“It doesn't matter, I want someone who can test”
“What tools do they need to have used?”
“It doesn't matter, I want someone who can test”
A pause ensues.......
“OK, so what qualities do you need?”
(Methinks: Ah, we are getting somewhere!)
So I explain that I want someone who has the ability to think creatively, be able to apply testing techniques to a testing problem, be a good bug finder, etc, etc.
“Oh, you want a junior tester, then?”
“Aaaarrrgggghhh”
I know what the real problem is with this conversation, it is that HR don't understand what testing is all about. They understand the need for technologies, tools, etc, that classifies the “type of person”. They can't understand the qualities required of a tester – or, I can't explain it well enough.
End result is that HR put the job spec that I have put together into their infernal machine that determines what the salary should be for this new hire and it comes out ridiculously low for an experienced tester, in fact it is at a junior tester level.
So, what do I do? I put together another job spec that has the requirements stated in terms of tools, technologies, methodologies, etc., re-submit that and lo and behold I get a salary level that looks about right.
The end result is that I have an advert out that states things I don't need as the only way I can get a decent salary level for the person.
That then perpetuates into the job agencies who filter out anyone who doesn't fit my “requirements” and, yes, I know I am potentially missing getting in someone who would be a good tester.
And then, when I interview people I am sure they wonder why I am not very interested in their skills that I ask for in the job advert.
Wish I knew how to resolve this problem....
Secrets of a Buccaneer Scholar - book review
I have just finished reading “Secrets of a Buccaneer Scholar”.
I ordered the book on May and it was delivered last Thursday. I put it onto my pile of unread and part read books and there it was destined to stay for a few weeks until I got round to it. Then something happened on Monday, which was a public holiday on the UK. I got sick. Not serious, just a bad cold. Unable to do the jobs I wanted to do in the garden, I looked around for something to do and I naturally gravitated to my pile of books. I had just started “how we test software at Microsoft”, but couldn’t get enthused about picking that up again. I almost started having another attempt at reading a book on how to speak Hindi, but that seemed too hard, and my book on Ballroom dancing techniques just requires too much concentration (I am learning to teach dancing). Buccaneer looked to be just about the right size, so I started it. I finished it in two chunks of time which is unusual for me as I often start a book and finish it weeks later (I am reading Weinberg on Writing: the Fieldstone method – which is a great book, but it is taking me time to get through it in small nibbles).
Getting back to Buccaneer. It is written by James Bach and is not a testing book, it is a book about how he learns things. He learns things differently to how I was taught to learn, he learns through his own method and it works for him. I was fascinated, partly because it works for him but also because some of the things he does and says struck a chord with me. For example, there is a great story in the book about how clams helped him to write more of the book (read the book, it will make sense!) because of his use of procrastination. I procrastinate too, I start writing/doing something and then stop, do something else and come back to the first task and do it better and/or finish it. But, I always feel guilty about the procrastination - “Finish what you started” was a mantra drilled into me from an early age. James does it almost by design and maybe in the future I won't feel so guilty.
The book has lots of stories in it from James' life and he uses the stories to good effect to make his points. He also uses mnemonics, SACKED SCOWS will remain with me now as something to work through. He uses heuristics which make sense, although I can see that they will require practice to make them work for me.
The very personal nature of some of the stories made the book come alive for me and it is easy to see how the messages can be used in a practical way.
Having read some of Jerry Weinberg's books, I can see his influence a great deal in James' book and that is not a bad thing nor is it surprising when you know that Jerry is James' mentor.
Not all of the tips and techniques in the book will work for me, I believe, but I am going to give at least some of them a try. I need to think about the book for a little while and then I will re-read it.
In summary, I loved the book and would heartily recommend it to anyone who is passionate about learning.
Well done, James, and thanks for a) taking my mind of feeling ill and b) giving me something I can use – I can't think of higher praise for a book than saying I will use what is in it.
I got a new Netbook!
I have just bought myself a new Netbook computer! A Samsung N120 if anyone is interested. Why? Well, I do a lot of travelling and a laptop is just too heavy, the battery life is just insufficient and it is too big to use on your lap on the train. And, I just don't seem to have to time to write for fun (as I am doing now) when I am at work or at home. This little beauty is just right, up to 11 hours battery life (or so Which? said) and nice and light for carting about.
Like any good tester, I didn't bother reading the manual before starting it up, I just booted it and made a start. Keyboard nice and responsive, touchpad really nice and has a scroll ability, like a mouse wheel (never seen that before on a touchpad).
All good so far, connection to my wireless network at home was simple, as it should be, set up user account for my wife and I, configured Mcafee, etc, etc. I then downloaded OpenOffice. I had used it before, but only briefly, and decided to try it out as I had heard some good things about it. It installed quickly, easily and I was soon able to write my first document – this one. I have to say, I am impressed, especially for something that is free. I have played with Calc, Draw and Base and will play, sorry, work with the others.
One major gripe with my purchase. I bought the machine based on the Which? report on netbooks and that is fine. I also bought mobile broadband, also based on a Which? report. I bought this from British Telecom and every day now I receive an email from them saying that delivery has slipped another day. I am starting to get fed up and considering going to another supplier.
Well, not really a testing blog entry, but I had to start somewhere. Hopefully, this means I will be able to write a bit more often into my blog now!
Oh yes, and the machine has a Webcam! No use whatsoever for me!
I did a bad thing last week
Last week I attended the BCS SIGIST (no, that isn’t the bad thing!). As usual, the day was well spent and enjoyable – Michael Bolton was excellent, as expected.
One of the presenters was Lloyd Roden. Lloyd is a very good speaker, I have heard speak him a few times and I like to listen to him. Lloyd belongs to Grove Consultants and I have enormous respect for Grove and the people in it, having recommended them to a number of people and I will continue to recommend them. His presentation this time was on his top ten controversies in Software testing and it was very entertaining, Lloyd is good at that. I agreed with his view on 8.5 of his top ten. The one I had a major disagreement was on certifications. Lloyds view is that they are useful, I disagree. That is not a bad thing that I disagree, however, I became incensed by his argument, interrupted rudely and argued with him whilst he was doing the presentation – that was the bad thing. I should have talked to him off-line, not interrupted and argued during his presentation, especially not argued with him whilst annoyed, as I was. Lloyd did a very good job of shutting me up politely, well done to him.
I have emailed Lloyd with an apology, and let this be a public apology.
What got me annoyed?
Lloyd argued that other professions use certifications, and he gave examples, therefore we should too. He also said that ISEB/ISQTB are good because they give common terminology. There are fallacies in this argument, as I see it, as follows:
Other professions do use certifications, it is true. There is a big difference, however, between the examples given and what we have in ISEB. The big difference is that professional certifications, like CORGI (now the Gas Safe Register), which was one of Lloyd’s examples, require the person who is certified to not only undergo training, but to show their competency. They can also have their work inspected to show that they do good, safe, work. From my own life, as I have mentioned before in this blog, I enjoy Ballroom and Latin American dancing (see http://www.sqablogs.com/petenairn/783/Dancing+is+like+Testing.html). I am currently training to be a Dance Teacher. I will (hopefully) get certified and to get that certification, I need to know the theory but also to demonstrate that I can actually dance as part of getting my certification. I cannot even study to become a teacher until I have passed other tests that show I can dance to a reasonable level. These are measures of competency and knowledge
What do you have to do with ISEB? Turn up to a testing station, answer some multiple choice questions and hey presto, you are a certified tester! Yes, you can go on a training course and even if you don’t then you will have to do some self study. But, you do not have to show you can test. This certification is, therefore, a measure of knowledge, not competency. See my previous entry for where the two can become confused http://www.sqablogs.com/petenairn/2296/Certifications+don%26%2339%3Bt+give+competency.html
To be fair to Lloyd, he made it clear that you can be a good tester with no certification and you can be a bad tester and have certification, which I agree with. So, I have to ask, what is the value of having the certification in that case? I find it difficult to respect a certificate if someone who has never tested software can pass the exam – and I personally know of two people who have done just that. If I know of two people, how many other people have also passed the exam who have never tested? How is that certificate valuable?
Is there a way of measuring the competency of a tester by the use of an independent body? This is where it is very difficult for a national or international certificate to be devised. I don’t have an answer to it, except to say that when I was at IBM, we had an internal certification program whereby you had to show competency before getting that certificate. It worked reasonably well, I was on the board that examined testers request for certification and we were very stringent in making sure we understood whether the person was competent enough to get that certificate. Certification wasn’t done by the use of a test, but by examining the actual work that the tester had done on one or more projects. That has its flaws too and I am not convinced in my own mind that the flaws outweighed the benefits, but it is a better measure than only sitting a test.
The real problem with the certification of testers, is the view that companies and recruiters take with respect to the certificate. I see a number of job adverts with certification as a requirement to even apply. This tells me that that the industry ascribes more weight to the certificate than it deserves.
Being able to apply for jobs is the only reason I got the certificate! Does that make me a hypocrite? Possibly, probably, but feeding my family is more important than principles to me. I wonder how many other testers took the test for the same reason? Is that a good reason for having a certificate? It appears to me to be the only valid reason I can find, but it isn’t a good reason.
Turning to the point about common terminology, which is another argument for doing certification, this is also fallacious. Common terms exist, but what people mean by them is different in the context of the company, project, test manager involved, culture, etc, etc. I don’t believe we will ever all agree on what is “common” because it always varies, even if slightly, depending on what you are doing at the time. In some professions common terminology is vital, e.g. if one surgeon calls an organ a “heart”, it is useful if the assistants on the operating table know what is meant. If we were all doing the same project at the same time, then common terminology would be important, but software development isn’t like that.
One other point I would like to make is that there is a difference between getting the certification and getting the training. I have heard the argument that certifications are a good way for consultancies and training companies to make money out of training courses. This may be true for some organisations where they only teach people to pass the exam. Some organisations, however, provide education on testing and also teach people how to pass the exam. The former are doing the industry no good, the latter are definitely performing a good service. I put Grove in the latter category – I have not been on one of their training courses, but people I know and respect have and from what they tell me and from the course materials I have seen, the courses are good. And, no, I do not have any affiliation to Grove whatsoever!
The arguments both for and against certifications can get emotional (as I did at SIGIST); I suspect the arguments will continue for some time and there will not be a successful conclusion for either viewpoint.
Last point from Lloyd’s presentation was the other 0.5 I disagreed with. He stated that Test Managers should set aside time each week to test. I think this is something we managers should do, but only if it makes sense on the project. When I have over 50 testers to manage, my time is better spent on managing than doing. By “better” I mean better for the project, better for the company, better for me and better for the testers – I am paid to manage, not to execute tests. I will expand on this in a future blog entry.
So, once again, sorry, Lloyd, for my behaviour – I won’t do it again. Shall we agree to disagree?
Certifications don't give competency
There are a lot of people arguing for and against certifications for testing. Personally, I don’t like certifications as I don’t believe they tell me anything about a tester’s ability to do the job. Here is an example of where certifications can give a tester a false sense of security.
My customer has sent all of the User Acceptance Team onto the ISEB foundation course and they have all taken the exam. Some have passed with no problem and some have failed, a couple more than once.
I was doing some fairly basic testing with one of the UAT testers the other day. We were testing out a change to some data rules whereby the data was shown to the user, or not, dependent on dates and class of data. I drove the system for a while and then he did. When he was testing, I noticed that he was using dates that were way earlier or later than the boundary at which the rule kicked in. I asked him why he had chosen those dates and he said that he was using Equivalence Partitioning such that any date would do. OK (ish) except that the rule was very boundary dependent and the better test would be to chose dates on, just before and just after the boundary. He also was testing every class of the data. Again, I asked why and he said he needed to check every one. Here was a good case of where equivalence partitioning made sense as the groups of classes behaved the same way. I had some difficulty in persuading him that different values would be a more effective way of testing the changes as he had convinced himself in his own mind that what he was doing was right – after all, he had been on the course and had been trained in how to use these techniques, had passed the exam and therefore he knew what he was doing.
So, although the tester had understood the concepts of Boundary Value Analysis and Equivalence Partitioning he did not understand the applicability in a real world situation.
It confirmed, in my mind, that the value of doing the training and taking the certification was low.
This is the problem with a knowledge-based exam/certification, it has no bearing on the competency of the person who has taken the exam. Competency can be trained, but not by the use of certification training.
Embarrassing moment...
One of the things I try to impress upon my testers is that when testing it is important not to “prove” that the system works correctly, but to find the weaknesses, faults, bugs. If a tester tests looking for the system to do what it is supposed to do, then that is what they will find – nothing (or very little), whereas if they are testing to find faults they are more likely to find them.
It is too easy to think “that is what the specification says, therefore, that is what I will test against”.
I fell into that trap this week. We have a lot of faults that have come back from Development and we have a short timeframe to re-test them. All of the faults are allocated out to team members to retest. One of my testers needed to complete some other work before starting on the re-tests and so I agreed that work needed finishing first, then I said “And then you can close off those faults”. After a short pause, he said “Don’t you mean that I should find the faults in the fixes?”. It was said in a jokey sort of way, but it was a good point!
Two things hit me. First, of course that is what I should have said and it was embarrassing that I had fallen into the trap. Secondly, I was very pleased that the tester had pulled me up on what I said, it means I am getting the mindset of the testers better.
Isn’t it odd though how even seasoned professionals like me can make that type of mistake?
Some FACTS about testing
FACTS:
- The customer is always right
- Automated testing always gives the right answer
- I don’t believe in coincidence
Maybe these facts aren’t correct?
Here is a cautionary tale of something that happened to us over the last two weeks.
We have a process with our customer in order to update some reference data on the database periodically. This data gets updated maybe a four or five time a year, data is amended or added to (never deleted). This is their data, spe******ed information that, quite frankly, we don’t claim to understand nor do we need to (hold that thought!). Over the years we have made this a pretty slick process; the customer provides a spreadsheet with the additions and the amendments which gets folded into the data on the database. We, the test team, have an automated script which compares what the customer wanted with what we have on the database. The automated script takes minutes to run (for those interested, this is a VB script in Excel). If all matches, we sign it off. Up until last week, we have had no problems with this in over 4 years.
Last week, we found a discrepancy when the automated script was run. Two values did not match. No big deal, a bug report was written and the database team corrected the data in the database, we closed off the bug report as the automated script now said that the two matched.
End of story…..
Well, not exactly. The changes went into the UAT environment, the users started using it and the system started behaving very strangely in a key application. By coincidence, there was a fault raised in Live at the same time on this same area. Aha! Duplicate fault from UAT in Live – not a problem, not too serious, don’t panic chaps! Upon investigation, the Live fault was found as being a user error so that’s OK then….
Well, not exactly. To their credit, the UAT team stuck to their guns and insisted that the problem in their environment was analysed. We sighed, grumbled about picky customer, you know the sort of thing, but we started investigation. It quickly became apparent that the problem that the automated script had seen was not a problem in the database at all, but a problem with the spreadsheet that the customer had given us. The database had, therefore, been incorrectly “fixed” to be equally incorrect. Causing the problems in the UAT environment. Also, this showed that the problem was not the same as in Live.
No problem! Uncorrect the database and everything will be fine…
Well, not exactly. The change was backed out of the database and the UAT environment was still behaving strangely. There was much scratching of heads, tut-tutting, sage experts poring over complicated SQL and even more complicated COBOL programs. After some considerable, stressful, hours, the problem was found. The misbehaviour due to the incorrect values in the database had caused flags to be reset so that when the correct values were restored, these flags were still set incorrectly. We corrected these and everything was OK – with 5 minutes to spare before the customer pulled the plug on an important release.
Debunking the facts!
- The customer is always right. Not really, the customer can make mistakes too. The real fault is believing that the customer has NOT made a mistake and jumping to the conclusion that the system is at fault. More thought required in diagnosis.
- Automated testing always gives the right answer. I never, ever, believed this one. Automation without thinking about the problem is not testing. It is, however, very easy to get complacent about an automated test that keeps giving you answers that look correct (discrepancy or no discrepancy). We got complacent about this process of updating data. We won’t make that mistake again AND we need to understand more about what this data does so that we don’t rely on UAT to determine whether it is right or not!
- I don’t believe in coincidence. Here, there was a real coincidence that threw us off the scent for some time. Coincidences DO happen!
Some interesting lessons learned (or, maybe, re-learned).
King Midas' ears - a rant
You know sometimes you just need to rant. I was reminded of King Midas’ barber and Midas’ donkey ears today
From Wikipedia:
“Once Pan had the audacity to compare his music with that of Apollo, and to challenge Apollo, the god of the lyre, to a trial of skill (also see Marsyas). Tmolus, the mountain-god, was chosen as umpire. Pan blew on his pipes, and with his rustic melody gave great satisfaction to himself and his faithful follower, Midas, who happened to be present.
Then Apollo struck the strings of his lyre. Tmolus at once awarded the victory to Apollo, and all but Midas agreed with the judgment. He dissented, and questioned the justice of the award.
Apollo would not suffer such a depraved pair of ears any longer, and caused them to become the ears of a donkey.[15] The myth is illustrated by two paintings "Apollo and Marsyas" by Palma il Giovane (1544-1628), one depicting the scene before, and one after the punishment.
Midas was mortified at this mishap. He attempted to hide his misfortune with an ample turban or headdress. But his hairdresser of course knew the secret. He was told not to mention it. He could not keep the secret; so he went out into the meadow, dug a hole in the ground, whispered the story into it, and covered the hole up. A thick bed of reeds sprang up in the meadow, and began whispering the story and saying "King Midas has a donkey's ears."
Today I feel like King Midas’ barber – I can’t keep this in any longer.
Something that really makes me cross is when so-called industry experts believe that they know better than you how to solve your problems. You see it in response to questions in forums, you see it in people’s blogs, you hear it at conferences and you hear it in conversations.
An example: I was at a conference and in the evening got into a conversation with one of the speakers. This speaker (who I will not name) is well known in the testing world, has written articles and books, teaches and speaks and is well respected. We talked a bit about generalities of testing and then got talking about what I was doing and his only retort was “I’ll give you my card, I would sort it out”. How could he possibly know he could sort it out? Why did he think he could do a better job than I could? He didn’t know what the context was, the constraints or anything about the project. The arrogance just stunned me and I couldn’t think of a suitable response.
My particular problems are split into a number of categories:
1) Those that are imposed as constraints on me from higher up. Things like the project’s budget, company policy. I can’t solve those, I have to live with them and make the best of them.
2) Those that are caused by the project methodology. The project uses a Prince 2-like methodology, which means that there are some things you just have to do. Whether you think Prince 2 is good, bad or indifferent is irrelevant, as with any methodology, problems are caused by it. Some I can work around, some I can’t.
3) Those that are caused by people working on the project but not in the test team. They have their own priorities, ways of working, constraints, etc. Some of those may cause me problems. Some I can resolve, some I can’t.
4) Those that are caused by people within the test team. These are totally within my control and I can work towards resolving them (e.g. skills gaps).
5) Those that are caused by me. Much as it pains me to admit it, I am not perfect and I cause some of my own problems. I try to work towards self-improvement, but perfection is probably about 300 years away.
I am sure that I am not the only one with this set of problems. I am glad there are problems; otherwise I would be out of a job!
I don’t believe that any one person can solve all of these problems for me. I most definitely believe that anyone could come into the project with no knowledge of its history, culture, methods and sort out my problems. If I had a free hand, had a lot of time and a lot of money I might, just might, be able to solve some of the problems this project has, but that is not going to happen.
I ask for help, advice, like most people. I am not afraid to listen to people who have tried things and found them to work. I have tried some of these methods, some have worked, some haven’t, but no-one, I repeat, no-one, could tell me how to sort out the problems I have.
So my plea to the experts is “Don’t tell me you would know how to solve a problem, give me pointers to how I can solve the problem”
Finally, here is a counter-example to the one I gave above.
I spoke to Johanna Rothman at a conference and asked her a question which was related to the talk she had just given. She did not give me the answer, nor tell me how to solve it. What she did was ask me questions about the problem and made a couple of suggestions of how I might approach solving the problem. It was wonderful, I was able to take that away and work on solving the problem myself.
Rant over. The reeds have finished their story and can now die.
Small things please small minds
I am so EXCITED!! We got a new version of the bug tracker yesterday to replace our 8 year old version. The new version does so much more, it does things that previously I had to dump data out to Excel, it looks better – bliss! I have even spent my lunch break playing to see what else it will do for me.
Yes, there have been a few teething problems, I managed to crash it once and some things have had to be tweaked, but I am so happy with it.
I know, I shouldn’t be so excited about a tool, but I can’t help it. Boy, is my wife going to get so bored with me tonight when I tell her all the great things I can now do so much easier.
Can’t stop to write any more, got to go play with my new version….
Bug emotions - correction
I had a problem posting my last entry, it appears that copy and paste of a Word table doesn't work too well here. It looked fine on preview, but not once it was in there.
My only excuses are that it worked on my machine and I ran out of time to test it - Aaaargh, I am going back to my developer days!
Now re-posted, not as a table.
Bugs with emotions........
In my last entry, Joe commented on me assigning emotions to bugs. That got me thinking that sometimes we testers do assign characteristics to bugs which are, after all, inanimate. However, just for fun, I tried to come up with the type of emotions that bugs might have if they could feel. I am sure that others could improve on these and/or add more.
Acceptance A bug that accepts it will be found. Typically a bug that is obvious as soon as you login (or even one that prevents you from logging in!
Affection A bug that loves other bugs. Some bugs are gregarious and like to live together, these are generally found all in one place
Alertness A bug that is alert to being found. It shifts its position so that just as you think you have it nailed down, it moves somewhere else. Memory leaks are good examples.
Ambivalence A bug that doesn’t care whether it is found. Typically a bug that you come across by accident
Anger A bug that hits you whenever you get near it, e.g. the blue screen of death
Angst See Anxiety
Annoyance A bug that gets cross when it is found and causes another bug to be created once fixed
Anticipation A bug that expects to be found and looks forward to it. Typically a bug that keeps showing up throughout the application
Anxiety A bug that worries about being found. These bugs hide in the corner and have to be winkled out with specially designed tests
Apathy See Ambivalence
Awe A bug that is amazed you have found it. It pops up with unusual error messages like “Error: This should never happen!”
Boredom A bug that just gets fed up with being there. These bugs usually go away of their own volition in the next release
Calmness A bug that can’t get you excited. Typically, a cosmetic bug, such as a mis-spelling
Compassion A bug that cares about you. This type of bug gives you an indication of what the problem is with a sensible error message.
Confusion A bug that is not sure whether it exists or not. Typically this type of bug can cause endless arguments with developers
Contempt A bug that really thinks testers are lowly creatures. Usually manifests itself with incomprehensible techno-speak in the error message
Contentment A bug that is happy with its lot, it likes being there. Usually these bugs are difficult to eradicate as they just like being there.
Curiosity A bug that looks for different parts of the application to get into to see what sort of damage it can do there. This type of bug is often due to a continual bad coding practice throughout the application
Depression A bug that throws itself at you suicidally. Normally found early on in the testing cycle
Desire A bug that finds testers incredibly attractive. Every tester on the project keeps finding this bug, usually finds its way into the bug tracking system multiple times.
Disappointment A bug that is disappointed you didn’t find it. Usually shows itself in Live
Disgust A bug that finds testers disgusting. Generally gives a supercilious error message
Doubt A bug that doubts its cause. Could be one of a number of different reasons why this bug exists
Ecstasy A bug that is high on illegal substances.
Embarrassment A bug that is embarrassed to be found. Generally has error messages like “Sorry, you can’t do that”
Empathy A bug that feels for the tester. Typically shows itself after you have had your morning coffee and never last thing at night
Emptiness A bug that has no content. This type of bug is where functionality is missing that should be there.
Enthusiasm A bug that has to show itself spectacularly, typically a system crash
Envy A bug that wishes it could be like other bugs. Typically this is a bug that looks serious, but in reality is only a minor problem
Fanaticism A bug that goes round telling other bugs how to create mayhem. Typically a comms bug that goes round multiple systems.
Fear A bug that fears being found. A particularly difficult bug to find and has to be tempted out into the open
Frustration A bug that wishes it could be a working system Tries hard to work, so only shows itself intermittently
Gratification A bug not found until system in Live. This is a happy bug has defeated all attempts to find it and has finally achieved its aim of thwarting you
Gratitude A bug that thanks you for finding it. When it is found it shows you the position of other bugs, so you get multiple bugs shown on one screen
Grief A bug that is inconsolable at the death of one of its fellow bugs and gives up trying to hide. Found when a previous bug, now fixed, was hiding this bug
Guilt A bug that regrets being there. These bugs are keen to atone for their sins and are, therefore, easily fixed
Happiness Found during “Happy path” testing.
Hatred A bug that hates you is one that you think you have found, but keeps coming back to haunt you in every release
Hope A bug that hopes not to be found. This type of bug just sits there and waits. It may be an easy bug to find or a difficult one, but is always obvious
Hopelessness See Despair
Hostility A bug that shouts at you. Usually shows itself with an error message ALL IN CAPS
Humiliation A bug that is so ashamed at being found that this part of the system never displays any bugs ever again
Hysteria A bug that shouts and screams all over the application, causing all around to react with panic. Usually a shows as a system crash
Inspiration A bug that is as a result of some really clever coding brought about through an inspired developer….who got it wrong.
Jealousy A bug that shows the symptoms of another type of bug which is superior to it
Kindness A bug that is kind to the developer by being easy to fix
Loneliness A bug that is a hermit. This type of bug is isolated from the rest of the application in an area that has no other bugs
Love Another gregarious bug. Not only does this bug congregate with other bugs, it actively hangs on to them, so you find two, three or more all doing the same thing
Lust This is a bug that spawns other bugs. A prolific breeder which means both tester and developer are forever chasing its children
Nostalgia Bugs aren’t what they used to be. This bug bemoans the fact that it hasn’t been found by Grace Hopper
Panic This bug causes all the parts of the system to go into meltdown. Typically this is a tight loop that consumes all of the CPU
Patience This bug just takes its time. Often seen as a Performance bug
Pride A bug that thinks it is the best bug in the whole system. Often surfaces before a fall-over
Repentance A bug that feels bad about existing and once fixed, fixes a number of other bugs as a side effect
Resentment A bug that does not like to be found, such that once fixed it re-appears again and again
Shyness A bug that is really hard to find and only surfaces in rare circumstances
Suffering A bug that causes the entire system to suffer. Often found as a result of security testing
Surprise A bug that does not expect to exist. Typically a simple bug that should have been found in unit test
Wonder A bug that is amazed to exist and cannot understand how a developer could possibly have coded it that way
Old Dogs....New tricks.
My dog, Rusty, is getting old, he will be 14 years old next month and we have had him since he was 12 weeks old. He is a terrier cross with all the bad traits of the terrier, but all of the good ones too. He has always been “good” at killing; rats, mice, rabbits have all fallen victim to him over the years. He is now blind in one eye, mainly deaf and gets out of breath after playing for more than about 10 minutes, but he still enjoys life and otherwise is very healthy for his age. Recently, we have had a rabbit visiting our garden every day and Rusty tried a couple of times to catch it, but the rabbit was too quick for an aged dog. After the first couple of failures, Rusty gave up chasing the rabbit. The rabbit was eating my plants and despite putting down animal deterrent chemicals, sharp sticks, cages round valued plants, the **** thing kept eating the plants. It started to become more and more blasé about me, my wife or Rusty coming into the garden when it was there, lolloping off slowly rather than fleeing in panic. Rusty kept eyeing it up, but didn’t chase it. Last night, Rusty got it – the rabbit got too close, was too slow and too complacent. Part of me was delighted to get rid of the pesky thing, part of me was saddened at the death of such a marvellous creature. I got to thinking, however, did Rusty play the game of letting the rabbit get more and more complacent with the thought in his mind that sooner or later he would get it? Or, did he just see the opportunity and go for it? He is an intelligent dog. Yes, I know everyone thinks their dog/child/pet is the best/most intelligent/etc. but he really is intelligent, so maybe he did play the patient, long term game.
What has this got to do with testing? As often happens, my mind turned to the similarity between life and testing. Bug hunting is not just about going full blast as fast as you can hunting down those pesky bugs, sometimes that fails, sometimes we have to play the long term game. If we are suspicious of an area in the system, we may have to be patient, try a different strategy. Intermittent bugs are one area where this strategy is best employed. Patience is key to finding an intermittent bug, determining what are the conditions that cause the bug to appear, trying different strategies, minutely changing some parameters to see what might cause the bug. Finally, to complete the analogy, the bug becomes too complacent at not being found and we are able to catch it and kill it. Strangely, there is delight at finding the pesky thing, but some sadness over the death of such a marvellous creature.
Real life and testing sometimes are too close together! Or maybe I need to get a life and stop thinking about testing….
Why is it so difficult to find good testers?
Who would have believed that it would be so difficult to recruit decent test automation staff?
I have now been looking for someone to beef up my automation team for over 3 months and only just found someone who is worth hiring. I have not been looking for someone out of the ordinary; at least I don’t think so. I have been looking for someone who is experienced in test automation, minimum of 3 years, has written automation frameworks and who has a reasonable idea of testing principles. QTP experience was desirable, but the key attribute was ability to automate. Salary would not be too much of a problem for the right person.
I have had over 60 CVs to go through, about 20 telephone interviews and face to face interviews.
Firstly, the CVs. Many people putting themselves forward for automation roles have only used capture/replay. This surprised me as I had the, obviously mistaken, belief that the days of doing only capture/replay had gone years ago; it appears not and the old myth of capture/replay being the way to automate seems to be alive and well. And some of the salary requests were outrageous. Many CVs were from software testing consultancies or outsourcing companies and the level of knowledge displayed by these people was woeful which only went to strengthen my opinion that software testing consultancies and outsourcing companies mainly body-shop and are only interested in placing people not skills. But, I digress. Many CVs came from people who had just arrived in the UK from India, which has given me the impression that there is quite a few migrants coming this way; which was interesting but most of the skills were not up to the mark and the CVs were, generally, quite badly written – although, having said that, the person I am going to hire has only just come to the UK from India.
Secondly, the interviews. I find telephone interviews useful to weed out the obvious no-hopers and people who do not tell the whole truth on their CVs. Being an interviewee over the phone is as difficult as interviewing over the phone, but the number of people who just could not communicate over the phone was staggering. The ones I got in for face to face interviews faired not much better. I don’t much like tests in interviews, but for an automation role, a very brief test was given, to write a short VBscript to solve a common problem (finding one item from one array in another array, finding the maximum of three numbers) and I was amazed at how many either could not do it or got it hopelessly wrong – one candidate even wrote the “VBscript” in “C” and that was wrong! There were some that had blagged the telephone interview that soon came unstuck. Most had (or said they had) the ISEB foundation certificate and could not answer simple questions that ISEB poses.
Thirdly, the recruitment agencies. I was explicit in my requirements, I even provided some sample questions with expected answers for them to ask candidates before I even got to see the CV. However, the agencies persisted in sending totally unsuitable CVs and people who could not answer the sample questions – not one single agency has been good that we have used.
But the thing that really makes me think about this whole exercise is that there are a lot of people out there who don’t know how to do a testing job and/or an automation job. They have been working, most for over 3 years, doing a testing job. The skill level is, therefore, low. No wonder that some people think anyone can do testing. It saddens me and I wish I knew how we testing professionals could change that.
The Test Sat Nav
My wife has an interest in “New age” beliefs, some of which she believes in, some of which she is sceptical about. Me? I’m always sceptical, however, I try to be as open-minded as possible and see what I can gain from any beliefs. Last night, we watched a DVD called “Law of Attraction” which, if I boil down 1hr 50 minutes into a single sentence, is “if you want something, focus on it until you get it”. OK, I think I was taught that when I about 5 years old, but hey, maybe some people need a reminder.
One of the analogies used interested me from a testing perspective (yes, I am getting to testing eventually!). The analogy was of a Satellite Navigation system in a car. You tell the Sat Nav where you want to go (your goal), it works out the best route for you and then tells you how to get there. The premise in the DVD is that we all have messages being given to us by non-physical means to guide us to our goal and all we need to do is listen to these messages and we will get there and achieve our goal. Personally, I found this hard to stomach, but maybe that is just me; there are too many variables in life for this to make any sort of sense and do I believe that I have this non-visible, non-physical “presence” sat with me all the time trying to guide me as to what to do? No, I do not.
Getting back to having a goal, the Sat Nav and testing. I got to thinking; do we have one overriding goal when we are testing? Wouldn’t it be great if there was a system where we can plug in that goal and it will tell us how to get there? Call it the Test Sat Nav (TSN)
I got to thinking that probably we do have a goal and that goal is to complete the testing successfully. The problem with that statement is what does “successfully” mean? I would guess that every Test Manager has their own, slightly different, interpretation of what “successfully” means and every Project Manager has their own interpretation that will be different from the Test Manager and every customer will have their own interpretation that will be different from the Project Manager and the Test Manager. So if we can’t agree a common meaning for the goal, how can we ever achieve it? And, therefore, how could we ever design a TSN that would enable us to plug in that goal and come up with the means of getting there?
So, maybe the goal is wrong, maybe it needs to be better defined, indeed if I put into the Sat Nav that I want to go to
By this time, I am now wondering if we have a goal in testing at all that we can quantify, plan for and know we have met. My idea of the TSN looks dead before it has started, if we don’t know where we are going, how can we plan to get there?
Then, the “
Having decided that, the TSN is easy, all I need is a program that will predict the changes that will happen along the way, the diversions that will happen and be able to predict the judgement call. In fact, all I need is another New age belief, that of psychic prediction – I’ll get my wife to sort this out for me.
Am I "too nice"?
I went to an interview last week. I was looking forward to the interview as I had heard good things about the company and they knew what good testing was all about. It looked like a good opportunity and the project sounded interesting.
I liked the interviewer and I answered all the questions as honestly as I could. Most of the questions were situational, of the format “what would you do if….?” Which makes you think about what you would do based on your own experience.
At the end of the interview, the interviewer told me that I would not be getting the position. I asked why and was told that I was “too nice” for the role. I know a few people who would disagree with any sentence that had my name and the word “nice” in it, however, the interviewer explained that the role required someone who, in the interviewer’s words, was a “b*****d” due to the nature of the project and the personalities involved and I didn’t have the personality required. It became clear that the project had an aggressive character; there was a lot of political manoeuvring; conflict was the norm. A female friend of mine calls such projects TDD. No, not “Test Driven Development” but “Testosterone Driven Development”.
I left the interview somewhat disappointed and deflated. One the way home, I reflected on the answers I had given in the interview and tried to work out what answers the interviewer was looking for. I came up with some answers that maybe would have fit the bill and worked out that, had I given them, they would not have been a true reflection of the way I like to work. I finally had to agree with the interviewer, I was not the right person for the position and I would not have done a good job in it. Clearly, the interviewer was a good judge of character!
However, I was still non-plussed by the “too nice” tag. I have been called a few things as a Test Manager, but “too nice” has never been one of them! So, I examined the answers I had given and compared them with the ones that the interviewer wanted and I came to the conclusion (which I knew already) that I like to work in a co-operative environment where everyone is working as a team towards a common goal.
A co-operative environment does not mean that you are always nice to each other, hugs and kisses around the meeting room are not going to help the project, and I have my fair share of violent disagreements and arguments but they are almost always concerned with the project, not personalities. I firmly believe that the co-operative environment gets the job done with better quality, lower cost and quicker than in a non-co-operative environment. If you are all pulling together with the same aim and following the same agenda, things work better.
But, what do you do if you find yourself in an aggressive environment? It has happened to me as I am sure it has happened to a number of us. My first reaction is to be co-operative, even in that environment, and it can work, I have proved it. I have one particular example where I turned an aggressive environment into a co-operative one. OK, it took me some months and a lot of hard work, but I got there. If trying the co-operative approach does not work, then you may have to get aggressive yourself, but if it is not in your nature you will find it difficult – I find it difficult and can only keep it up for a short period of time.
So, going back to this position I didn’t get. A part of me wonders if I could have turned it around and made it into a co-operative environment. I will never know. The interviewer would not take the risk with me and in the interviewer’s place I wouldn’t have taken the risk with me either, the interviewer made exactly the right decision, in my opinion.
I’ll stick with believing in the co-operative environment, but I doubt I will ever hear anyone call me “too nice” again!
Buy my slogans!
I have found a way of making my millions. I have a problem with moles in my garden; they dig up my flower beds and leave hills in my lawn. I don’t like to kill the little creatures as all they are doing is what is natural to them and surely I can put up with a bit of inconvenience, it is not worth an animal’s life to stop that is it? But, I would like them to stop, so I want to deter them.
I have tried sonic devices that moles don’t like the sound of, I have tried scent related deterrents, but the following morning, there is another one or two mole hills in my lawn.
I have discovered a method of removing moles from my garden that is as effective as all the proprietary methods and is considerably cheaper. My method is to put posters around my garden with slogans saying “Moles keep out”, “Moles go home”, “This is a mole free garden”. Believe me when I tell you that this is as effective as anything you can buy, i.e. it is not effective at all – just like anything else you buy!! I am sure if I market this “solution” as well as all the other products I have spent a fortune on that haven’t worked, I could make millions.
Slogans are pointless, they do nothing to improve any situation. Around our office, the Account Manager had posters put up saying “Are you the key to the next release’s success” and “Do it right first time every time” and my all time hated poster “Are you at peace with your piece of the next release?” What on earth does anyone think these will do for the project?
I find it incredible that any reasonably intelligent human can think that a slogan on a poster will make a difference to another reasonably intelligent human being. It may work in a factory where the work is mind-blowingly boring and repetitive, I don’t know, but it will not work in a creative environment like an IT shop. I tackled the Project Director about these posters and he stated that they were intended to keep people focussed. Oh, that is alright then, so people would not be focussed if the posters were not there? Of course they would, all the posters have done is give some amusement and a lot of irritation, can’t you high-ups see that?
I tore down the two posters put into my area and put them in the recycling, see, I knew I would find a use for them! It is my job to motivate people and keep them focussed and I do not need posters to do it for me. I didn’t see any lack of focus afterwards.
By the way, if you want to see some posters that are the exact opposite, go to www.despair.com, they made me laugh.
Moles, however, are much less intelligent beings, they are sure to be affected by my posters and they will be deterred from my garden. Please email me with your order. I can accept cash, cheques and I am just about to set up a PayPal account. Only £2.00 per poster or 5 for £8.00 – you won’t regret it..
A consultant's tale
A few years ago, I went into a large multi-national bank to perform a consultancy task to review their testing practices. As a consultant, you get one of three types of reactions when you enter the building:
- A Groan – This is often because they have had bad experiences with consultants before and think this one will be no different. I don’t mind a groan, this often tells me that they have decent practices already and you won’t find much to worry about. This type of assignment is generally praising what they have done and recommending improvements in some areas.
- A Hurrah – This is the juicy assignment. A hurrah usually means that the workers know that what they are doing is insufficient and are hoping that the consultant will finally tell the bosses that they need to get their act together. This assignment often means that there are no (or very bad) practices and you are starting from ground zero.
- No reaction – These are the worrying and most hateful assignments. This often means that someone high up has called in a consultant with no buy-in from the staff and they are not going to co-operate with a single thing you do. You have an up-hill battle to even find a desk or get told where the toilets are. Your best bet is to do what you can, write the report and get out.
Anyway, back to this bank. I got the “Hurrah” reaction when I walked in. I soon found out why. In the section I had been called into, there was a development team of about 80 and 2 testers, yes, count them, 1, 2. These 2 testers were from one of the high street branches and had been called in a couple of years previously to give the system a quick once over before it went live. 2 years later, the system was still a mess. Some releases had gone to live and some of it even worked, but the system had some chronic problems.
I started doing the usual things for gathering information and it quickly became clear that the main problem was not only the testing (surprise!) but the Project Managers. One Project Manager, during my discussion with him stated “We incentivise our staff with a bonus which gets more the fewer bugs are found in the code”. I asked him how the tester was incentivised and he replied with a puzzled look on his face “The same, of course”. So, the testers got a greater bonus the fewer bugs they found!
The testers had been given no training in testing, they were both intelligent people but they were not IT people, they were business people. I asked them for their test plans and test scripts and it came as no surprise to me that I got a 1 page test plan and 2 pages of test script.
The reaction to the presentation I gave to the senior bosses was extremely positive. You could almost hear the sighs of relief around the room that the problems had been identified and that an action plan had been proposed. This came as a surprise to me as I was expecting some hostility (I had pulled no punches). It turned out that Director had not been with the bank for very long, which I knew, and he had been haranguing the senior managers about the lack of quality and they just didn’t know what to do about it – now they had a way forward.
That 4 week assignment to review their testing practices turned into a 4 year stint with them as they asked me to put the action plan in place. And a very happy 4 years they were for me. And I kept on the two business people, trained them and they became very effective business testers.
The lessons I learnt from this experience were:
· Just because it is a large company, it doesn’t mean they have good practices
· Senior IT managers do not always understand how to improve quality, they need telling.
· The situation of lack of testing in an organisation is an opportunity, not a reason to be depressed.
· “Shooting from the hip” sometimes works!
{ Last Page } { Page 1 of 3 } { Next Page }
About Me
My Profile
Archives
Friends
My Photo Album