SilkTest Automation

Finding Maximum Text Field Size

Our application (and presumably yours too) has the occasional text field. Every text field has a maximum length defined in the specification (or it should). Automation makes it easy to verify these maximum lengths.

The first thing I did was to define a routine to test maximum text field length. I pass that routine the TextField in question, plus the expected length (as per the specification). If the length matches the expected length, no messages are posted. However, if they do not match, an error is logged, indicating the actual and expected lengths.

The first way I've found to determine the actual length is by a brute force approach. I wrote characters to the field (using TypeKeys), then read the field back to see how long it is. I do make sure I set the Agent's keyboard delay to zero first, and restore it to its original value when I'm done. This allows me to stuff in those keystrokes as fast as possible.

The sad thing with this approach is that it's quite slow for really large text fields. Most of our fields are under 64 characters long, and those go pretty fast. However, most text fields that do not have a maximum length set will accept 16,384 characters, and that generally takes more than an hour.

With a little creativity, I could make some changes that should speed it up. For instance, if my expected maximum were 80 characters, I could verify that 80 works, and that 81 fails (i.e. returns 80). This would let me try just two tests - the expected length, and that length plus one. This would give me a quick pass/fail response. If it does fail, however, it would be nice to know what the maximum is set to, so I can pass that on to the developers when I write up a QA ticket.

There are several methods to find long lengths. Right now, I add on one character at a time. It would be easy enough, however, to add on 100 at a time (or any other number you like), then keep adding until it fails. At that point, subtract your increment and starting adding one at a time (or ten, or whatever). This method should cut down  the number of comparisons by an order of magnitude or more.

The simplest method would be to stuff a string of 16,384 characters in with SetText, read it back with GetText, and get the length of the return string - this should be the maximum length. This would cut the number of SetText/GetText calls to exactly one. Now, this routine takes the same amount of time no matter how long the maximum string is.

Followup: I now have a two-part test, and this seems to work best. First, I check the anticipated length plus one - if the anticipated length equals the real length, then we're done with one short SetText/GetText call pair. If that fails, however, I attempt to stuff in all 16383 characters (the maximum string length in SilkTest). This takes longer than the first test but is only needed when the string is not the correct length. Now, the result is this test takes less than a minute for the longest strings, and only a few seconds if it actually matches.

Builds - How Often?

Where I work, we normally get a new release every week to test. That was pretty good, in a way, because each person should be able to get through all their test cases in a week. Generally, about half of Build Day was spent just installing the new bits. My typical installation included one client, one server, and about ten other add-on features, each with their own installation program and potential problems.

That's half a day, per week, just to keep my system current. And it doesn't include the database or sample data...if I have to recreate all of that (i.e. clean install), then we're looking at at least a full day. For just one server. Our app can run with two distinct servers (one with an embedded database, one that uses an external database). The database possibilities include several flavors of Oracle, SQL Server, DB2, and others. We also have server bits that can run on AIX or Unix/Linux. Our web component can use almost any web server made. Finally, the whole system is available in multiple languages.

Starting to feel overwhelmed? Well, we're not through yet. Servers can be Windows 2000, 2003, or very soon 2008. 32 and 64 bit. Clients could be Windows 2000, XP, or Vista, again in both 32 and 64 bits as appropriate. And to truly test the right way, I'll need those clients in every supported language. Oh, and don't forget, we support several major releases of our product, going back at least several years.

Now you might be thinking that one build per month might be more appropriate, but instead of that, they went to one build per day as the application became more feature-complete. It's good in a way - bug fixes appear more often, which can occasionally be critical. It's bad too, because I can't spend all my time updating the software.

For my automation development, I have settled on a happy medium...I grab a build every Monday, and again on Wednesday. I can automate the actual installs and they might then be able to do a single client/server set in an hour or so, depending on the speed of the hardware.

I can take this a step further, and will be sometime this year. We're moving to some massive multi-core servers and VMWare's Lab Manager. This will allow a substantial percentage of our test configurations to be available virtually, with all the benefits that entails. Setting it up will be a bear, but using it on the other end should allow automation to fully test the application in almost every possible configuration in an amazingly short time. I'm looking forward to the challenge.

Sort Order

As I may have mentioned before, our software is available in English, German, Spanish, and French. I need to write my tests so they run in all supported languages, and along the way need to test the actual text strings themselves. Part of that testing involves sort order.

Many of the things we display in lists have a list header that can be used to sort the list on different columns. Many of my test cases have me validate that clicking on any particular list column header will indeed sort by that column, and sort correctly.

So, what exactly is a "correct" sort? It seems the correct answer is "it depends". The next question is, "How can I determine if a list is sorted correctly?"

According to the SilkTest help file, there are only two sorts available in SilkTest - ArraySort and ListSort. As you might imagine, they sort arrays and lists. Since I'm testing the sort order of lists, it makes sense to use the ListSort. The sort is said to be a "dictionary sort", which I would expect in this case, but makes no mention of exactly what the dictionary is nor whether it can be changed. It does mention this sort uses the Windows "locale" setting.

The actual code to determine if a list is sorted is pretty easy - I take the original list (from the GetContents(iColumn) method), make a copy of it, sort the copy, then compare the two. If they are identical, then the list is sorted. However, this only sort-of works.

First problem - upper versus lower case. An "a" doesn't sort near an "A", but rather all lower-case alphas sort after the upper-case alphas. Yet, in a phone book or dictionary sort, case is irrelevant. Sadly, SilkTest makes no provision for setting the case sensitivity. I work around this issue by upper-casing the list items before I make the sort copy. And I should stress it is a workaround, not a fix.

Next comes the issue of special characters...where should they sort? I'm only concerned with the special characters available from the keyboard, such as the shift-number keys. Unfortunately, it appears these characters sort in odd ways - some are before the numeric (0-9) characters, some between the numeric and the alphas, and some after the alphas. In my mind, it seems they should all sort before the numerics, or after the alphas.

Finally, there's the problem of accented characters. For instance, Spanish has the Ñ character, while French has accented vowels and German has vowels with umlauts as well as the ß character. Where should they sort?

In an ideal world, they should sort next to the matching unaccented character (or N or S for the other two), but only in the language that uses them. For instance, the Ñ should sort after the N in Spanish, but after all the alphas in English, French, and German (since it's not part of their language character set).

I'm still working on this, and will probably need to create my own custom sort routine, where I can apply a different dictionary for each language. Both our application, and SilkTest, claim to sort based on the Windows locale settings, but as near as I can tell, neither one does.

I'll have further details if and when I solve this one.

Localization Testing

My current job has an application that is available in four languages. A previous employer had applications that were available in even more languages. We have created an easy method of dealing with localized interfaces.

Basically, the various text strings used in the program are stored in an INI file. I use the IniFileGetValue and IniFileSetValue calls to move these strings to and from the INI file.

To start with, record all your window definitions with ID numbers, text, and index numbers enabled. Then edit that window definition to remove most text. I keep the window tag text (i.e., from the title bar) and augment it with the tags for the other supported languages. The window tag is the only location that keeps the text - all other control declarations use the index (# tag) or ID number ($ tag). You'll need to record the window declaration in each language to assemble all of the tags.

All other tags come from the INI file. I started this process by writing a routine (which I call ValidateUIString), which is passed a "key" and a language identifier, and returns a text string. The key is a text value which indicates where the string in question is located. Example keys may be "BUTTONADD" for the Add button, or "MENUFILE" for the File menu. The language identifiers are EN, SP, DE, and FR, which are the identifiers Windows uses for the different languages. These map to English, Spanish, German (Deutsch) and French. We have a routine that can (by opening the Help/About dialog) detect which language we are running in. We can also detect the language at the time we login to the application.

The ValidateUIString routine does two things. If there is a string matching the key and language, it returns it. If there is no matching string, it creates one in the INI file. Therefore, the first pass thru any test would store the strings in the file, while all subsequent passes compare the string against the value in the file. This offloads the actual "file creation" task to the code, which is a much better solution than forcing a human to enter all the strings. Alternately, it may be possible to write a program that grabs the strings from a resource file (i.e., software source code) and creates the INI file from that.

This routine can detect several possible errors. The obvious one is that the strings do not match - this is logged as an Error. Another possibility is that the string is not there - in this case the string is written to the file and a Warning is logged. One other possibility is that the returned string is truncated - this error happens due to a bug in SilkTest. It's routine allows me to write a long string to the file, but when reading it back, it only returns the first 256 characters. So, I make sure to check for this condition and log it as a Warning.

The next task is to write a test case, or (in my case) a sequence of test cases, that bring up all possible dialogs and check their text. What do I check? Well, all static text fields, all button text, list headers (which are often sort buttons as well), and the contents of most drop-down lists and popup lists.

When these test cases are first run, all text is added to the INI file. You'll need to run these cases on all of your supported languages to get the correct text strings. All subsequent runs will check the text against this "baseline" and inform you if anything changes.

At first glance, you might wonder just how this would help your testing. Yes, at some point, you have to trust the translators and their results. But many errors can easily be detected. One type of error happens when buttons have a typo in their captions. For instance, the button saying "Add..." may be correct in most dialogs, but you might find one where it's spelled Add with no dots after it. Since all Add buttons should say the same thing, this one is pretty easy to find and fix.

Another area where you might find errors is in the length of the strings. For instance, if any string that consists of one word in English is found to consist of multiple words in any other language, you should verify that translation.

A lot of these errors could be found just by eyeballing the INI file. For instance, if a multi-word entry in English looks like a "valid sentance", with a leading uppercase letter and an ending period, then chances are good that all other languages should also have a leading uppercase and trailing period. Of course, this can vary by language too, so it helps to have a feel for the languages you support.

This subject requires some depth, so in future entries here I'll give specifics about the construction of the INI file and how to use and maintain it. This is one of the areas where SilkTest's versatility makes it my automation tool of choice.