SilkTest Automation

Builds - How Often?

Where I work, we normally get a new release every week to test. That was pretty good, in a way, because each person should be able to get through all their test cases in a week. Generally, about half of Build Day was spent just installing the new bits. My typical installation included one client, one server, and about ten other add-on features, each with their own installation program and potential problems.

That's half a day, per week, just to keep my system current. And it doesn't include the database or sample data...if I have to recreate all of that (i.e. clean install), then we're looking at at least a full day. For just one server. Our app can run with two distinct servers (one with an embedded database, one that uses an external database). The database possibilities include several flavors of Oracle, SQL Server, DB2, and others. We also have server bits that can run on AIX or Unix/Linux. Our web component can use almost any web server made. Finally, the whole system is available in multiple languages.

Starting to feel overwhelmed? Well, we're not through yet. Servers can be Windows 2000, 2003, or very soon 2008. 32 and 64 bit. Clients could be Windows 2000, XP, or Vista, again in both 32 and 64 bits as appropriate. And to truly test the right way, I'll need those clients in every supported language. Oh, and don't forget, we support several major releases of our product, going back at least several years.

Now you might be thinking that one build per month might be more appropriate, but instead of that, they went to one build per day as the application became more feature-complete. It's good in a way - bug fixes appear more often, which can occasionally be critical. It's bad too, because I can't spend all my time updating the software.

For my automation development, I have settled on a happy medium...I grab a build every Monday, and again on Wednesday. I can automate the actual installs and they might then be able to do a single client/server set in an hour or so, depending on the speed of the hardware.

I can take this a step further, and will be sometime this year. We're moving to some massive multi-core servers and VMWare's Lab Manager. This will allow a substantial percentage of our test configurations to be available virtually, with all the benefits that entails. Setting it up will be a bear, but using it on the other end should allow automation to fully test the application in almost every possible configuration in an amazingly short time. I'm looking forward to the challenge.