FACTS:

- The customer is always right

- Automated testing always gives the right answer

- I don’t believe in coincidence

Maybe these facts aren’t correct?

Here is a cautionary tale of something that happened to us over the last two weeks.

We have a process with our customer in order to update some reference data on the database periodically.  This data gets updated maybe a four or five time a year, data is amended or added to (never deleted).  This is their data, spe******ed information that, quite frankly, we don’t claim to understand nor do we need to (hold that thought!). Over the years we have made this a pretty slick process; the customer provides a spreadsheet with the additions and the amendments which gets folded into the data on the database.  We, the test team, have an automated script which compares what the customer wanted with what we have on the database.  The automated script takes minutes to run (for those interested, this is a VB script in Excel).  If all matches, we sign it off.  Up until last week, we have had no problems with this in over 4 years.

Last week, we found a discrepancy when the automated script was run.  Two values did not match.  No big deal, a bug report was written and the database team corrected the data in the database, we closed off the bug report as the automated script now said that the two matched.

End of story…..

Well, not exactly.  The changes went into the UAT environment, the users started using it and the system started behaving very strangely in a key application.  By coincidence, there was a fault raised in Live at the same time on this same area.  Aha!  Duplicate fault from UAT in Live – not a problem, not too serious, don’t panic chaps!  Upon investigation, the Live fault was found as being a user error so that’s OK then….

Well, not exactly.  To their credit, the UAT team stuck to their guns and insisted that the problem in their environment was analysed.  We sighed, grumbled about picky customer, you know the sort of thing, but we started investigation.  It quickly became apparent that the problem that the automated script had seen was not a problem in the database at all, but a problem with the spreadsheet that the customer had given us.  The database had, therefore, been incorrectly “fixed” to be equally incorrect.  Causing the problems in the UAT environment.  Also, this showed that the problem was not the same as in Live.

No problem!  Uncorrect the database and everything will be fine…

Well, not exactly.  The change was backed out of the database and the UAT environment was still behaving strangely.  There was much scratching of heads, tut-tutting, sage experts poring over complicated SQL and even more complicated COBOL programs.  After some considerable, stressful, hours, the problem was found.  The misbehaviour due to the incorrect values in the database had caused flags to be reset so that when the correct values were restored, these flags were still set incorrectly.  We corrected these and everything was OK – with 5 minutes to spare before the customer pulled the plug on an important release.

Debunking the facts!

- The customer is always right.  Not really, the customer can make mistakes too.  The real fault is believing that the customer has NOT made a mistake and jumping to the conclusion that the system is at fault.  More thought required in diagnosis.

- Automated testing always gives the right answer.  I never, ever, believed this one.  Automation without thinking about the problem is not testing.  It is, however, very easy to get complacent about an automated test that keeps giving you answers that look correct (discrepancy or no discrepancy).  We got complacent about this process of updating data.  We won’t make that mistake again AND we need to understand more about what this data does so that we don’t rely on UAT to determine whether it is right or not!

- I don’t believe in coincidence.  Here, there was a real coincidence that threw us off the scent for some time.  Coincidences DO happen!

Some interesting lessons learned (or, maybe, re-learned).