So the exciting part of testing is the analysis and planning ?

Yup, the quality has nothing to do with the programmers and everyone else involved so I wasn't falling for that - and certainly not for the very low salary that was being offered

So 20 years of working for the same company came to an end last week when I handed in my notice.

A touch of sadness that my struggle to get some professional testing going had come to nothing but when you are waking up at 4am with arguments about why testing should be done properly going through your head then it's time to quit.

A bottom-up approach to change can only go so far, especially when surrounded by apathy and/or resistance from the majority and you get a slapping down from the CEO

I wasn't learning anything and felt myself being dragged down into the mindless click testing that everyone else was doing, the job I'd had to struggle to get into was starting to turn sour

Took me awhile to get a new position ( a few blogs on THAT coming up soon ) but then I found a job ad that ended with

No, I haven't got a job with The Colonel tasting his new recipes.

I have though found one of the problems that can happen when testing the same program for a while - repetition.

Not the boredom of seeing the same dialogs day after day but the problem that you can end up using the program the same way every time - when creating a user you always enter the address first and then their job and then you go to the next screen where they are assigned a function.

I realised I was doing this when I was doing some regression testing and followed the steps to reproduce and found a bug before I'd even got to testing the original bug. The steps to reproduce had me doing things in a different order than I'd got into the habit of doing.

The next realisation of this occurred when we had some customers in for a demonstration from the sales guys. The customers are well looked after and provided with a tasty lunch but they mustn't have been too hungry as there was some leftover for us to pick at.

Two chicken pieces on skewers in hand I went back to my testing - and with one hand holding the chicken I had to use the keyboard to drive the app under test and not the mouse..

and voila

a bug

Fortunately the bug was not in the chicken but in the code

So more learning experience for me:

1) vary the steps you take to test and I'll try to get another person involved as well to supply a fresh pair of eyes and approach to the tests

2) remember to include keyboard driven testing as part of the test plan

How do you get past the repetition problem ?

A long time without blogging as I've been busy

I was assigned to another project to help out and do a small programming enhacement ( ignoring the Mythical Man Month advice that throwing extra staff at a project doesn't help ). Played around with the program to get familiar with how it worked and soon had it crashing. And crashing. And crashing

Not good

Looked at the spec for the enhancement I was meant to be doing and found lots of gaps in it as well as a huge hole which meant it wouldn't actually work.

So I was able to spend my time testing the program instead of programming which has been great.

I wrote a very basic test plan so I was able to focus my testing efforts and could also give the boss some idea of progress but I did warn him that the more I tested the more ideas I was getting for testing.

There was almost a proper release cycle set up as well - a release would be made and I'd do a smoke test, bugs reported as fixed in that release would be tested and closed (or not) and I was even running some regression tests which proved to be well worthwhile as it picked up some old bugs coming back.

However this meant that my time spent testing the area that I was supposed to be testing went down though it did mean the program as a whole was moving forwards and not backwards.

At the last project meeting the boss asked if extra testing resources would help.

Sure they would

Except by testing resources he means the woman that writes the documentation and the support guy that does the training. Neither of them are professional testers or trained testers or even think like testers.

Would he ask them to help out with the programming as a programming resource ?

No

So despite all the Quality talk of the last few months it still seems that we're not walking the walk

Not only can databases store stuff but the way people use them - or not - can be instructive.

The programmers on the latest project for example. They intend to just use the one.

So how are they going to test how their application behaves when it's misconfigured and there's 0 of the things they are expecting ? Or how the interface looks when there's many many items to be displayed ?

Does it behave correctly when there's just one option and the use doesn't have a choice to make ? How will they check their errors that are meant to appear when a user gets halfway along the path and finds they can't go any further as there's no data to populate the next screen ?

Or are they just intending to pass it over to the testers and tell them that it works fine when everything is happy happy ?

As part of my coding duties I'm still responsible for a program that configures a system. I had to make a small enhancement that meant adding two more fields to a table in the database and changing a dialog so it could read and write to these new fields

Made the program and passed it over to our 'tester'

Waited for him to ask me which database fields the enhancement used...

and waited...

No request for information arrived.

So I knew he wasn't looking in the database to see if the program was writing to the correct fields, he was just checking that the program was consistent with itself. If I'd made a mistake and was writing the Max value to the minimum field and the Min value to the maximum field then as long as I made the same mistake in reading the values then the program would appear to work.

Until the program that uses this data comes along and tries to use it...

I had a constructive afternoon writing a number of SQL scripts that would populate a database with one, two or many values. Mix the scripts up and I could have a large variety of differently configured systems very easily and be all set to test.

Now that is a sign that things are improving here - having control of the data we are testing with and not just using any old customer database that happens to be lying around and could be full of any old cr8p

Before the quarterly Divisional Meeting I sent an email to the Sales Director and asked him, if our company was a car company, which one would it be ?

Honda - known for reliablity

Audi - known for technology ( or at least a technical slogan ! )

Mercedes - quality at a price

As he drives one I shouldn't have been that surprised when he answered Mercedes.

Do Mercedes send you 3 engines in a day before one works ?

Do you hire a team of mechanics to check your Mercedes over for a week before you drive it ?

Sometime in the future we'll ship software to our customers that they don't find bugs in within an hour.

Sometime in the future our customers will realise they don't need their own large test teams as the software we send them just works

Then we might consider ourselves a Mercedes

Theres a city in my mind
Come along and take that ride
And its all right, baby, its all right
And its very far away
But its growing day by day
They can tell you what to do
But theyll make a fool of you
And its all right, baby, its all right

Were on a road to nowhere

Talking Heads

Those were the words from the lead developer that totally floored me for a while.

"They're just bugs" he'd said "I don't know what you're getting so worked up about them for. It was a one line fix for this one " he continued, pointing at one of the ones I was worried about " why are you getting so excited about them ? "

This was at a meeting ( Gasp #1 - we had a meeting between programmers and testers ) to discuss a number of issues, one of which was a number of points found by my exploratory testing ( Gasp #2 - exploratory testing had been done months before we actually shipped an EXE )

I was taken aback by this response so only gave a weak reply that this bug might not have been found until Sprint 6 or even until it was out at a customer.

"Well we would have fixed it then" was the response "I've already said it was a one line fix so why the big fuss ? "

It took me a while to think of a suitable response but I think I have it.

1) The developer has contradicted himself. He started off the meeting by saying how the company couldn't carry on doing what it has been doing which is shipping untested software to a customer on a Monday morning, having bugs reported Monday afternoon, shipping a fix on Tuesday and the whole sorry cycle carries on for 2 years.

So he does care about bugs and wants them found in-house rather than by a customer. He should be very happy that there's someone who does care that there are bugs in the program.

2) It was a one-line fix for the particular bug he pointed out. So they got lucky with that one and it didn't mean a major rewrite of a whole class module or function. The next bug on the list might not be such an easy fix.

3) If they're not bothered about bugs now, when will they be ? By Sprint 6 when there is a backlog of 500 ? In which case they are in the same situation that he wants to avoid getting into.

4) If they are not aware of the bugs now then they will be putting new code on the top of buggy code. Why not have a solid foundation rather than trying to build on quicksand ?

"It's Just a Bug" - well according to Micahel Braidy, one of the Hallmarks of a Great Tester is that they are

Excited by bugs

A great tester thinks bugs are cool. A great tester shows up in a developer's office on a regular basis with a big grin eager to show off the latest nifty keen horridly awful bug that the tester found in the developer's code. A great tester boasts about bugs to other testers and eagerly listens to other testers' exploits

One of the new programmers notices my screen flashing away as Watir exercises the application.

I am sitting back, hands nowhere near the keyboard or mouse, the screen is flashing away.

He wanders over to ask what I'm doing and how am I doing it.

So I explain about Ruby and Watir and show him some simple scripts and explain how I'm using them, ask him what his experience of testing is and how were his programs tested at his previous job.

Just another little nudge to getting the T word used and thought about

With the arrival of my new machine, my programming tasks out of the way ( which the new machine helps with - just 1 minute to compile the program instead of 30 minutes ) , I was all set to rock n roll on the new project.

Didn't take long to find some problems so I rewarded myself by having a play with Ruby and Watir to see what they could do.

Just took a few minutes before I could sit back and watch as a new user was automatically created, followed by 20 others. Even more fun when I set the speed to fast and watched the screens flash by.

Cool

Not just cool but potentially very useful. With a little bit more work I'll be able to test out all the places where a user can provide input by driving them with Watir. We'd already written some Perl scripts that hit the web service directly, now we can do it via the browser itself.

Create an input file with the combinations you want to test ( Perl or Ruby are great for this ), fire up the script, look for errors, then read the database ( Perl or Ruby again ) and see if what got into the database matches what you were trying to put in there.

Very cool.

I then played around some more and wrote another script that created a user, logged out then back in again, visited every screen in the application ( only a few at the moment ) and performed an action.

Again, very easy and quick to do.

Showed the programmers so they could see what the brave new world of testing could do - they loved the bit where it went speedy and the forms flashed by.

But the significance of it escaped them and they didn't quite get how it would remove the tedium of manually checking the inputs to every box on the screen.

They just wanted to know if I could run lots of browsers and overload the system

A few minutes later I had it doing so. The programmers then complained that I was always getting the same session ID so wasn't really thrashing the program.

The next day I looked into the problem some more. Altered my script so that each browser was being created in a seperate thread but still the session ID's seemed to be the same

OK then, I'll try creating browsers manually - set the default home page to be the start page of the app and got my fingers double-clicking

aha

AHA

Doing it manually I was occasionally getting the same session ID

Got the programmer over who said it was impossible and that I was just creating a new window and not a new browser

Got him to watch as I clicked

And there it was in front of his eyes

So programmer excuse #2 was trotted out - yes it's known problem and it's Microsofts fault and we cant do anything about it.

I then wrote another script which started new browsers up after a time interval that I could set when kicking off the script and found out that if a new browser was started less than 2 seconds after another then sometimes it would get the same session ID.

Would I have found this problem without a tool like Watir ?

Possibly.

But it made it real easy to do so - and narrow down when found.