2007-Nov-23 - Lessons learned from the last project

Hooray! Another project is coming to the release day next week!

Some lessons can be learned from the last phase of the project, especially some points I missed in my test design and execution, which resulted in some bugs were not found in earlier phase.

1. Checkpoints of product name change.

Our product name has been changed during beta phase due to legal issues. Here is the checklist of the possible places contained product name:

1) File name, including shortcut name.

2) Registry

3) Process name

4) Service name

5) Windows title (web/win32)

6) Text in message boxes, web console, command line, and other UI.

7) All documents (Guide, readme, online-help, usage prompted in command line, etc.)

8) Items added in OS settings. For example, Windows firewall exception list, including program name and service name in advanced settings.

#8 is what I've missed in my test.

2. For executable file name changed, check all documents, including user's guide, readme, online-help, usage prompted in command line, etc.

I missed to check the usage help. It used the old name of the program.

3. Test special characters according to the file storage format.

For example, if a value is to be written to an .ini file with the following format:

#comment

[section]

param = value

Then test special characters like #, [, = in the input value. These cases are effective.

In my test, when I input "a=b" as the value, the program returned "param = a", which ignored the part after "=" in the value.

4. Execute with one single argument a time.

If a program can be run by multiple arguments, there must be cases that test each argument (standalone) at a time.

For example, if program ABC has 3 arguments a1, a2, & a3. Then MUST test "ABC a1", "ABC a2", "ABC a3" respectively.

Check out the lesson I learned: I only tested "ABC a1 a3" and "ABC a2 a3", and found both cases passed, then I assumed all 3 arguments were working. But finally in the last moment I found actually "ABC a1" and "ABC a2" failed. Only a3 was working. This bug should be found much earlier if test each argument alone first.

Of course, cases combination of arguments are required too.