Originally posted on my main blog

Test automation, it turns out, is a broad term. The tools with which to do it range from the massive, expensive, infrastructure intensive, to the open source download-and-run. The latter are easy to deploy and make quick use of in an agile environment, but even in this context some suggest the best outcome is achieved by assigning dedicated resources to it.

Inherent in our culture is the constant striving to find ways to do everything better. We tend to jump to technology for this, assuming that if we can do it, we should, be it food production, human reproduction, national defense, or of course, software testing. But should we always look to technology to make things better? Catapulting headlong into uncharted technological waters just because we can has been known to backfire. Consider DDT , DES , the atomic bomb. We’ve paid a high price to learn that in many situations it’s healthier to do things “the old fashioned way”, rather than act on the assumption that technology could somehow do them better.

It strikes me that the way Exploratory Testing is now defined is exactly how we tested when I was just starting in the field, before we attempted to formally define it, and before we decided we could test software better by writing more software to test it with. Automation seems like a positive (and harmless) technological advance because it makes highly efficient work of crawling through software, performing in mere hours several human-days’ worth of keystrokes and verifications. But its inherent limitations can place at risk precisely that which it’s designed to protect and improve. Automation only tests what we tell it to test. It lacks the human intuition that tells us when to stray from the path to look in a new corner, peel back a curtain, or lift up a rock. These are places bugs like to hide. Like the consumer of DDT-laden produce, software tested primarily by automation can end up sicker than it might have had it enjoyed the magic touch of old fashioned exploratory testers.

The question is not so much can we use automation in an agile environment, but should we? Certainly there is a place for automation in software testing, but it can’t do it all. We should use it with care lest it become the costly and potentially caustic stuff caked on otherwise healthy organic software testing.

Originally posted on my main blog

Recently, I did a Google search on “agile test automation”. I’m struggling to figure out how, if at all, test automation fits in to a very agile and loosely structured software development model. I come from a world of three- to twelve-month software release cycles. Where I work now, we release new code to production at least once a week. It’s exciting and invigorating. I have a substantial background in writing test automation – not record-playback, but object-oriented, extensible code whose creation and maintenance is the full-time job of automation engineers. What I most enjoyed about writing test automation was the challenge of building in extensibility. Without extensibility, automation is one-dimensional: it does one test or set of tests and reports what it finds. In agile development, software changes in small and not so small ways almost constantly. As the software changes, simplistic automation can’t change with it (at least not without more human intervention), and as such has limited value. By the time we make the test code robust enough to add sustained value, the product it’s designed to test has probably morphed two or three times over. Is robust automation being fast-paced into obsolescence, and if so, is there a new, nimble automation approach to replace it? Or is my search string an oxymoron?

In pursuing these questions, I stumbled upon articles slightly off topic but decidedly more interesting, about the mysterious process we call “exploratory testing”. This is the farthest thing from automation. It involves creative thinking about how you'll approach the software, some knowledge of the target market, a notebook to record findings, and a chunk of uninterrupted time. In addition to these basics, skilled testers bring to the table some knowledge of the risk areas, and my favorite, intuition. The thrill of the hunt is on: you don't know exactly what you're looking for, but you'll know it when you find it.

You may have heard software testing referred to as an art. I was delighted to discover Jonathan Kohl’s article likening exploratory testing to music. He offers that key both to successful software testing and great music is the proper balance of tension and resolution. Without this balance, he says, neither testing nor music is at its best, and both are boring. James Bach defines exploratory testing as “simultaneous learning, test design, and test execution.” Remove the word “test” from this definition, and it sounds a whole lot like musical improvisation. Testing -- like music -- as art. Nice. I agree with my test junkie colleagues that exploratory testing, rather than “don’t color outside the lines” scripted testing, keeps us engaged and nets better results: more bug finds, in shorter test sessions.

So the search didn’t lead to what I thought I was looking for. I still haven’t satisfied my question about the place of automation in agile development. But, like exploratory testing, following the unintended path led to the richer, more interesting discovery of like-minded thinkers in the field.