mr toad's wild ride

Where have you been, Mr Nairn?

I noticed with shock that it is over 2 years since I last posted a blog entry.  Why?

It is interesting that I have not stopped writing, just stopped posting here.  I looked at what I have written and each of the potential entries are:-

- Not complete, either in thinking, writing or both

- Internal to the company

- Dreadful!

Partly my enthusiasm for posting stuff sort of went away due to a number of factors.  In recent months I have re-discovered my enthusiasm and have started using my twitter account.  I  think I will start blogging again. 

So, I will make this a short entry and hopefully get posting again more regularly.

One of the things that has got me more enthused recently (and I will probably blog on why my enthusiasm waned) was that I have been seeing more and more enthusiasm within my own test team in my new company and that has been a long hard slog.

I have been having some very interesting testing discussions with one tester in particular that has made me think hard and I always like to do that.  Some of the results of those discussions will make great blog entries!

And so it goes

Over the last week or so the computer industry has lost two giants/innovators.  First being Steve Jobs.  Nuff said on that one.  Second is Dennis Ritchie, co-creator of the C programming language and UNIX.  I think the following comment by a reader of an article about Dennis Ritchie is an appropriate RIP for him:

My life on the D-List (speakers and known consultants)

At times I feel like Kathy Griffen, living my professional life on the D-list as a speaker and "industry" consultant.  Most of it is my own self imposed position on the fringe.  I am outspoken and opinionated, but try to only be so when it is really needed. 

I've seen too many "experts" with their blog sites spouting and ranting on about the same thing as other people.  There is a lot of rehash of topics and ideas in our industry right now.  But every once in a while there is something new.  It just seems that Testing has kind of stagnated again in its progress as a profession, at least that is my opinion.  The same crop of "experts" have been at for the last 10 years or so, and some of their messages are getting old too.

So what does this have to do with my life on the D-list?  Not much except a feeling I've had lately about wanting to get more involved in the industry, and I'm doing it to a small degree.  I'm speaking this next week at STPCon 2011 in Nashville, so if you are going please stop by and say hello.  Maybe even listen to my presentation, and be good enough to give me feedback so I can make it better.

I'm thinking about doing some more writing and publishing, but that takes time.  And right now I'm looking for my next gig, I still have bills to pay.  Trust me I have ideas for a couple of articles, one of which may be a bit of a handgrenade being thrown into the automation debate. 

So... I'll see you around.  And who knows by this time next year I might have moved up, to the C-list that is.

Testers are like Fighter Pilots

If you think about it our "mission" is to go out and find the enemy (defects) and shoot them down (or at least be the air tactical guys who guide the fighter pilots to them).  We do this by various means and record our victories with symbols (defect reports) on our planes fuselage.

Now I don't know about you, but my plane is starting to look like a pile of flying post it notes.  How about you, how many "kills" do you have?  How many times over are you an Ace?

Traditional vs. Agile methods

Because I am a smartaleck... In tribute to George Carlin (think Football vs. Baseball)

Traditional vs. Agile

----------------------------

Stern Manly voice: In Traditional methods we drive to production.

Not so manly voice: In Agile methods we iterate until it's good enough. 

Stern Manly voice: In Traditional methods we push a system out the door.

Not so manly voice: In Agile methods we deliver a demonstrable piece.

Stern Manly voice: In traditional we have firm and well defined requirements in documents in a repository.

Not so manly voice: In Agile we have loosely written user stories on post cards on a wall.

Stern Manly voice: In traditional the Project Manager is the quaterback of the group and guides the team to the release date.

Not so manly voice: In Agile we are all equal on the team and run together through the sprint.

Stern Manly voice: In traditional we have specifically trained people who wear a specific hat.

Not so manly voice: In Agile we have cross-trained people who can wear multiple different colored hats.

Stern Manly voice: In traditional we meet on a predifined weekly schedule to discuss the progress of the work.

Not so manly voice: In Agile we meet every day for 15 minutes to talk about what's blocking us.

Stern Manly voice: In traditional the product is the most important deliverable in the project regardless of individual contribution.

Not so manly voice: In Agile the people are the most important thing because we want them to feel good about their product.

New idea for a new year

A friend of mine just told me about an interesting philosophical axiom.  This was a translation from russian (he is a Russian ex-pat, now U.S. citizen) he relayed to me, and I think it fitting for a new year.

"It doesn't matter which hand a monkey holds a grenade with." 

Think about it a little.

It is only a one line fix

We had a problem report outstanding for a long time, simple little problem. When a user gets a message in their inbox, a yellow marker shows up on their screen to indicate they have a message waiting. Some messages are important messages that the user has to action and then they get a red marker on their screen. Some users get the important messages as a FYI, but were still getting the red marker when they can't action the message and in this circumstance they should receive neither a red or yellow marker. Low severity and low priority bug.

Fix comes in from Development. The fix is to remove the marker for these users, which is a simple one line fix.

We test it. It is a low severity/low priority bug amongst all the other bug fixes we have to retest, to do a lot of testing in this area requires a lot of data set-up, so we just do a quick check. All is well.

The release goes to Live.

Everyone reading this knows what is going to happen next, otherwise there would be no point to this post! Yes, something was broken. True the users who were supposed not get their marker didn't get their marker, however, no other users were getting any marker either and this marker is relied on for important messages, as the users need to take action - no action means the user is prevented from using the system. Not getting the marker meant some users were not actioning and getting locked out of the system.

Big “oops”.

OK, so all test groups make this type of error sooner or later and we learn from them (sort of, sometimes we don't). But the whole episode made me think about Michael Bolton's distinction between “checking” and “testing”. What we tend to do on low severity bug retests are “checks”. Heck, we have already tested this area once, lets just check the fix. We might do some regression testing, maybe through running an automated suite, but we rarely test a bug fix that is low severity.. Maybe other test groups do more testing on this type of bug fix, but mostly my group doesn't.

So, I asked myself, SHOULD we have done more testing? In order to answer that, I looked back at all the low severity bugs we had found, over 3000 of them. I looked at how many of these fixes had either failed retest and/or caused a problem in Live. Few had failed retest, less than 1 percent. I could only find one other low severity problem that had caused a problem in Live and that was also low severity. I also estimated what would it have taken to test these fixes rather than check them? A lot was the answer. The rate of return for putting testing effort into these bug fixes would have been incredibly low.

So, my conclusion was that, yes it was bad that the bug got through, but I don't want to start testing low severity bugs, it will cost too much effort that would be better targeted at more important areas finding more important bugs. I will just have to take the risk that another low severity bug fix will cause a problem in Live.

Testing is all about assessing risks and acting accordingly.

Problem with hiring testers Pt2

We are in a recession, the unemployment rate is increasing, so why, oh why, am I finding that it is so difficult to find a good tester to hire?

I have some theories as to why there are so few people applying for my vacancy, but no real facts.

And the CVs I have received are, quite frankly, dreadful and those that aren't dreadful, when I interview it is clear that the CV lied. An example, one CV said that the person was experienced in SQL and in the interview the person could not describe how to do a simple SELECT statement! OK, SQL experience is not vital to be a good tester, but if you say you know some SQL, then please, please, be able to describe what you use SQL for and how to do simple operations.

Do applicants really think that the person interviewing them knows less than them and that they won't be caught out? I guess they must and it makes me wonder whether applicants do get through some interviews and start work doing a poor job? Probably yes, especially as these people are in work now.

It all got me thinking about the state of the testing world (again, I sometimes get thinking this way!) I am surprised at the lack of skill in testers who have 3,4,5 years “experience” and yet there are a lot of testers in the software world. I am baffled. Does this mean that there are a number of companies who have test teams made up of poorly skilled people? Does this mean that there a number of companies who are paying people relatively high salaries to do a bad job?

Now, maybe I am lucky, the company I work for values testing as an integral part of the development of software and is keen to employ good testers and every company I have ever worked for in a testing capacity has had the same view, so maybe I just haven't worked for a company where this isn't so and there are really test teams out there who are useless? Thinking about it, I did go for a job once with a company that clearly had no clue about testing – they wanted to hire me as a Test Manager and the IT Director interviewed me (very badly, as I recall) and it was clear that I would be a bit part player in the decision making process. I turned the job down. Maybe I should have taken it just to see how I could have changed things.

I know I have been accused of being picky in the past when hiring testers, and I would agree, I am picky – because I want someone who is skilled in the craft of testing or someone who has the potential to be skilled. Surely that should not be too much to ask?

This week I have, finally, filled the vacancy. Assuming he accepts the offer. It has taken me about 6 months to find him.

Automation is like an Ogre is like an Onion. It's got layers!

A quick post here about some things I've read lately about test automation.  Specifically the test execution methods and the different levels in software that it can be implemented against.

With Agile methods there is a heavy emphasis in TDD (Test Driven Development) to use Unit Testing and the harnesses that support it.  This is good, and about time for development to do some testing and automation of their own. 

Next we get into using these harnesses to create Integration level tests and to incorporate them into the build process as part of Continous Integration (CI).  Again, a good idea whose time has long been due.

Now we get to the higher level tools, harnesses and methods for doing the API and "below the surface" tests (as some have called getting under the 'skin').  This adds another layer of testing we can do on the system and help to improve the efficiency and effectiveness of the testing via automation.  I'm all for that.

Finally we get to the GUI layer (where most of the test automation work is still done today) and all the fun we have with that.

So as you can see there are different layers of the software that we can peel away to see and test via automation.  Hopefully some of those layers don't have too strong a smell and make your eyes water up.  Just like when you peel an onion.  And as part of that you cannot judge the software by its outer layer alone, you will need to do some peeling further down to make sure things really work the way they are supposed to.  This is the complexity issue we deal with in software in general.  Nowadays there is a lot of complexity in our systems.

At times this complexity makes us look at our systems as being like a big green Ogre, it just gets ugly.  But if we spend the time and effort to go beyond the surface we find it isn't all that bad.

Really... really.

I'll follow up another time with more details and findings about the various layers. Right now it is a bit late and I need my beauty sleep.  ;-)

The problem with hiring testers

Tony Bruce asked a question on Linkedin that I just had to blog about.

Tony was asking why job adverts are so specific in their requirements for testers instead of asking for a good tester.

I think this is a good question and one that I have been struggling with for years. Let me give an example of what has been happening to me recently when trying to recruit, so from the other side of how Tony was looking at it.

What I have been looking to recruit is someone who can test. Full stop, that is what I want. I don't care if they have a background in banking, health care, games, whatever. I don't care if they have used QTP, Selenium, Robot or any other tool. I don't care if they have been using Agile, Waterfall or any other methodology. I don't care if they have Unix, Windows, mainframe or any other knowledge. I WANT SOMEONE WHO CAN TEST!!

So, I go to HR and say that. I get blank looks at first, then I explain a bit further and the conversation goes something like this:

“Oh, you want a junior tester, then?”

“No, I want an experienced tester who knows how to test.”

“So what technologies do they need to have?”

“It doesn't matter, I want someone who can test”

“What tools do they need to have used?”

“It doesn't matter, I want someone who can test”

A pause ensues.......

“OK, so what qualities do you need?”

(Methinks: Ah, we are getting somewhere!)

So I explain that I want someone who has the ability to think creatively, be able to apply testing techniques to a testing problem, be a good bug finder, etc, etc.

“Oh, you want a junior tester, then?”

“Aaaarrrgggghhh”

I know what the real problem is with this conversation, it is that HR don't understand what testing is all about. They understand the need for technologies, tools, etc, that classifies the “type of person”. They can't understand the qualities required of a tester – or, I can't explain it well enough.

End result is that HR put the job spec that I have put together into their infernal machine that determines what the salary should be for this new hire and it comes out ridiculously low for an experienced tester, in fact it is at a junior tester level.

So, what do I do? I put together another job spec that has the requirements stated in terms of tools, technologies, methodologies, etc., re-submit that and lo and behold I get a salary level that looks about right.

The end result is that I have an advert out that states things I don't need as the only way I can get a decent salary level for the person.

That then perpetuates into the job agencies who filter out anyone who doesn't fit my “requirements” and, yes, I know I am potentially missing getting in someone who would be a good tester.

And then, when I interview people I am sure they wonder why I am not very interested in their skills that I ask for in the job advert.

Wish I knew how to resolve this problem....