Hooray! Another project is coming to the release day next week!

Some lessons can be learned from the last phase of the project, especially some points I missed in my test design and execution, which resulted in some bugs were not found in earlier phase.

1. Checkpoints of product name change.

Our product name has been changed during beta phase due to legal issues. Here is the checklist of the possible places contained product name:

1) File name, including shortcut name.

2) Registry

3) Process name

4) Service name

5) Windows title (web/win32)

6) Text in message boxes, web console, command line, and other UI.

7) All documents (Guide, readme, online-help, usage prompted in command line, etc.)

8) Items added in OS settings. For example, Windows firewall exception list, including program name and service name in advanced settings.

#8 is what I've missed in my test.

2. For executable file name changed, check all documents, including user's guide, readme, online-help, usage prompted in command line, etc.

I missed to check the usage help. It used the old name of the program.

3. Test special characters according to the file storage format.

For example, if a value is to be written to an .ini file with the following format:

#comment

[section]

param = value

Then test special characters like #, [, = in the input value. These cases are effective.

In my test, when I input "a=b" as the value, the program returned "param = a", which ignored the part after "=" in the value.

4. Execute with one single argument a time.

If a program can be run by multiple arguments, there must be cases that test each argument (standalone) at a time.

For example, if program ABC has 3 arguments a1, a2, & a3. Then MUST test "ABC a1", "ABC a2", "ABC a3" respectively.

Check out the lesson I learned: I only tested "ABC a1 a3" and "ABC a2 a3", and found both cases passed, then I assumed all 3 arguments were working. But finally in the last moment I found actually "ABC a1" and "ABC a2" failed. Only a3 was working. This bug should be found much earlier if test each argument alone first.

Of course, cases combination of arguments are required too.

When port web application to IIS, developer has to do the following conversion on his codes, so that the Unix files meet Windows convention. QA hence need to pay attention to these points too.

Feature	Unix Implementation	Change to Windows Implementation
Directory separator	Forward slash (/).	Backslash(＼)。 Note: In Perl script, backslash is used as an excape character. So you have to use double backslash(＼＼) to indicate the backslash in a directory.
File names and pathnames	Unix supports long file name and extension with 4 characters. File name is case sensitive.	FAT file system uses DOS 8.3 filename convention. And it's case insensitive. NTFS is case aware, yet case insensitive.
Directory hierarchy	Unix file system appears as a single directory layer.	Windows storage is seperated to several physical and virtual drives, each of which has a directory hierarchy. So you have to specify the drive letter(C,D,E,etc.) as part of the file path when access a Windows file.
Text file line termination	Unix only uses LF(line feed) to indicate the end of a line in text file.	Uses both CR(carriage return) and LF to terminate a line in text file, and some applications.
File desciptors/handles	Unix file descriptors 0, 1, and 2 indicate stdin, stdout, and stderr respectively.	Win32 uses handles instead of descriptors. Call Win32 API GetStdHandle() to confirm which handle is in use.

File path length limitation

Unix filename is restricted to 255 characters.

Windows file path is restricted to 259 characters.

I summarize the checkpoints when we test file path:

(A) Normal cases

1. The path is absolute path
2. Relative directory

(B) Force Error

1. The Directory is read only
2. The disk space is not enough
3. The path contains wildcard char

(C) Internationlization

1. Directory with special characters
2. The path contains other languages
3. The path contains DBCS name
4. The path contains DBCS and SBCS name

(D) Platform compatibility

1. The path is Unix path
2. The path is double slash--C:\\test\\log
3. The path is single and double mixing slash--D:\\test\log\\abc
4. The path contains Windows and Unix mixing slash
5. The path is Unix, Windows, single and double slash mixing--D:\aa\\bb/c//dd
6. The path is DOS 8.3 format

(E) Other considerations

1. The path is UNC path
2. The path is networking mapped driver 
3. The path is very very long
4. The path is hidden.

I read an article about what is quality and the side that we easily forget. It raises a good point about objective quality and subjective quality. Usually we disregard subjective quality and make bug-free, with lots of features, but customer hated products.

Why does the product with high objective quality still fail to satisfy customers? What should we focus on while testing to improve subjective side of product quality?

I remember another article about customer satisfaction. It says there are 4 levels of satisfaction, and the higher level must be based on all of the lower levels.

These 4 levels are:

1. Accurate.

    That means the product basic function must meet customer's requirement. Like a cell phone must have the function of dialing and answering a phone call.

2. Accessible.

    That means when customer wants to use the function, it can be instantly found and used. Like simply press a few numbers or just shout out a contact name, the phone call can be made.

3. Partnership.

    That means you have to let customer feel that you are on her side. Like if call center receives a phone call from a customer complaining that there's something wrong with her cell phone, call center should send out another cell phone for temporary replacement and ask the customer to send back the broken phone.

4. Advice.

    This is the highest level of customer satisfaction, that customer wants to learn. Like advising customer to use touch screen on a cell phone, which most customers do not expect to have this requirement.

We often talk about customer insight. Regarding the 4 levels of customer satisfaction, what a QA can control may be only the lower 2 levels. The higher 2 levels can hardly be TESTED, but are decided when requirement documents are made.

Obviously quality control starts from requirements analysis phase.  

is that it is not fully tested.

We spend most of the time thinking how to automate more cases. We spend the last minute coding on our automation tool to cover one more case. There's no time for us to test our program, and we have to use this quality-unknown thing to test our real product. Isn't it a BIG risk added on the product and schedule?

If we do this test automation framework again, I think we must have a clear plan. And one phase must be included in the plan is the CODE FREEZE and start TESTING phase. During this phase, no more features added to the framework, no more test cases covered, no matter how easy it seems. During this phase, the most important thing is to test whether it will give a PASS when ALL the inputs are correct, and a FAIL when ONE of the input is wrong.

I've been a QA engineer for 2 years, not including my nearly 1 year part-time tester experience when I was in college. I learned functional testing, localization testing, some system testing, some process, and some test automation. I like what I'm doing now and I think I am kind of good at it. But recently the manager has talked with me about career path, and I start thinking about it. Where should I go 5~10 years from now?  Should I stick with this career, or try something else?

What I am worrying about is whether this is a career for the youths only. Just look around me, most of the employees are in their 20s or 30s. Only very few in managerial positions are over 35. I don't see a clear path for a QA to follow. I even have considered going back to school to study another degree, like Intellectual Property, so that I can change profession.

But when I searched the internet (I searched a lot in sqaforums too), I realized that every career was difficult, but if you like it, it would be easier to go through. At least I like what I'm doing, I like designing test cases for an object, I like reporting the bugs I find, I like learning automation tools and scripts and use them to help me execute cases, and I like working with people in and out of local office. It's much risky to change the career, and the future is more uncertain. I think I will just expand my skills and go deeper in current career.

Although I'm still not sure what choices are for a QA's future besides QA manager. Someone listed senior QA, project manager, technical director, independent consultant, or owner of your own business. Perhaps consultant is a choice I'm interested. But I do not know the roadmap. I don't have acquaintance has that experience.     

Matt Clapham outlines the top 10 tips testers should know to securely test applications.

1, don't test as admin.

2, test on a secure file system. Test out other file systems, such as NTFS, FAT 32, FAT 16.

3, don't change hive key local machine in registry or program files. The best place to store user data is in a hive key current user. Somewhere under their user profile or in the application data folder.

4, don't store secrets as clear-text. Only store it if it's absolutely necessary. Store a hash of the secret.

5, get adequate feature documentation.  You need to know what registry keys the feature is using or the application is using. You need to know the DLLs that are involved. You need to know where the support and temporary files are stored, and what permissions are on those. And if it's an XML data transaction you need to know the XML schema, the data types, and the sizes. Similarly if it's binary data, see if you can get some de-bug hooks that will let you look inside of the data structures and understand more about them and be able to look in them and get back out those other details you're looking for. Also find out if the feature is on by default.

6, press the limits of bad data.

7, test on a dual-processor system.

8, know the 20 laws of security.

9, use the various tools at your disposal.

10, analyze the source code.

Matt's presentation on MSDN TV: http://msdn.microsoft.com/seminar/shared/asp/view.asp?url=/msdntv/episodes/en/20030429securitymc/manifest.xml&rate=1&WMPVer=10.0.0.4036

The transcripts: http://msdn.microsoft.com/seminar/shared/asp/transcript.asp?url=/msdntv/episodes/en/20030429securitymc/manifest.xml

Common checkpoints:

1. Data Encoding - ASCII, SBCS, DBCS, MBCS, Unicode, ...

2. Locale/Cultural awareness - date time format, currency format, number format, string parsing, ...

3. Text input - code page, font, layout, display, ...

Special points that need attention:

1. Strings - newly added text strings, tokens parsing, tokens with special character like %, ...

2. Dependant applications / softwares - test the server, client, or other applications in target locale language,  and the popularly used ones.

3. Special services for the target locale - the product name (may be different from English version),  Activation Code,  embeded URLs (eg. Update URL), ...

The problem found during a test: SMTP performance drops seriously while running volume test on both SMTP and POP3 of a gateway product.

Related modules: SMTP module, POP3 module, Scan module (which scans the network traffic)

Possible causes: Related modules problem (bug), data volume, specific test sample, others.

To narrow down the causes step by step: (--> Result  ==> Conclusion)
TEST 1: Load SMTP traffic only. No POP3 traffic at all.
--> Much higher SMTP performance.
==> POP3 traffic has impact on SMTP performance.

TEST 2: Load POP3 traffic only. No SMTP traffic at all.
--> POP3 performance does not have severe difference compared to the test with both SMTP and POP3 traffics.
==> SMTP traffic does not have impact on POP3 performance.

The following test try to remove the "others" cause:
TEST 3: Disable unrelated features/actions, eg. notification, agent, debug logs, transaction logs, etc. Load both SMTP and POP3 traffic.
--> SMTP performance still drops seriously. No better. Same result.
==> Exclude "others" cause.

The following tests try to remove the "data volume" cause:
TEST 4: Adjust POP3 traffic to 5 mails/connection and 1 mail/connection. Compare the results.
--> No better. The same.
==> Exclude POP3 traffic volume cause.

TEST 5: Minish POP3 sample mail size to range between 15KB ~ 26KB.
--> No better.
==> Exclude POP3 Scan volume cause.

The following test try to remove the "specific test sample" cause.
TEST 6: Replace all of the POP3 sample mails with SMTP sample mails.
--> No better.
==> Exclude "specific test sample" cause.
==> So now we have the "Related modules problem" left for tracking the cause.

TEST 7: Run the same test on the previous version of this product.
--> SMTP performance is better on old version.
==> It may be caused by the new features of this version.

TEST 8: Disable all of the POP3 Scan features and run the test on the current version again.
--> No better.
==> Exclude POP3 Scan features cause.

TEST 9: Disable all of the SMTP Scan features too, except for NRS (Network Reputation Service, a new feature which is only applicable to SMTP).
--> No better.
==> Exclude SMTP Scan old features cause.

TEST 10: Disable all Scan features including NRS.
--> SMTP performance increases to the old version level.
==> It is possibly caused by NRS!

TEST 11: Load SMTP traffic only and do the following tests for comparison:
11-1: NRS disabled.
--> High performance.
11-2: NRS enabled.
--> Performance drops a lot.
11-3: NRS enabled but add the ip addresses to the white list. 
--> Performance drops a little than 11-1 but better than 11-2.
==> NRS IS the cause of this problem!! (Or at least, one of the main causes.)

One question: how to explain the result of TEST 1 with the final conclusion?
Since NRS and other SMTP/POP3 scan features are all implemented by Scan module. When the traffic load is not heavy enough, Scan module can handle the tasks better. But when the traffic load becomes heavier, scan tasks are queued, and performance drops obviously. So it seems it's POP3 causes the problem but it's actually not. POP3 only increases traffic load and magnifies the symptom.

DQ: SELECT

DD: CREATE, DROP, ALTER

DM: INSERT, UPDATE, DELETE

DC: GRANT, REVOKE

After a bug fixed, or a feature added, QA not only verify the bug or new feature, but also should consider: 1) affected modules, 2) other affected QAs, 3) affected part of documents.

For example, say the QA who tests product registration found that the registration URL has changed. So not only (s)he has to test the product registration with the new URL again, but also:

1). Test the new url link in web console and installation wizard.

2). Tell the QAs who test web console and installation about this change.

3). Mark in somewhere that the old Administrator's Guide and Installation Guide (or Quick Started Guide) should include this update in Product Registration section, and Installation section, which has mentioned the registration URL.

This is a simple example, usually the changes are much bigger, especially between 2 versions.

The corresponding documentation changes are often missed consideration. I propose that an OVERALL review is still needed for new version documents, not just review the new feature part.

I have been mentoring some newcomers and interns. I have some points I'd like to highlight regarding mentoring.

The good thing we've done is the documentations prepared for the newcomers to survive in the company. Documentations include company policy, useful links for daily work, QA related, and project related.

But there're 2 things we should improve:

1. More emphasis on bug report title and format.

Although we have some guide for newcomers about how to write a bug report, when they begin to write their own, it's usually complained by the developers.

Standard:

The title template: Object + Action/Behavior + Condition.

Description format:

     Summary (must have) 

     Precondition (optional)

     Steps

     Expected result (must have)

     Result

     Additional info (optional)

Usually newcomers have different style of report titles. Some may use lots of abbreviations that are not understandable (in order to keep the title short). Some may create some code names that only the creator can understand.  That makes developers unable to understand what the report is about from the title, hence, they say, waste their time to click and open the report and read it through to get the idea. Not only for developers, poor report title also makes it difficult to query the case by keywords in title.

And for the description part, newcomers will not use judgmental words because we've told them not to do so. However, a common behavior of them is to SHOUT IN THE REPORT! ! So we should also add this avoidance in a bug report: avoid using all capital words or excalmatory mark.

2. Trace the execution of blocking test cases.

When some critical bugs block some test cases, usually for newcomers, especially inexperienced interns, they would just skip all these cases happily and never run them again even though the blocking bug is fixed, until the next test cycle.

I think we should add a field in the bug report, listing the ids of the blocked cases. Once the bug is fixed and verified, the listed cases should be executed immediately.

At the same time, in the test cases management system, if the test result of a case is NT (Not Tested), should add a comment about the report id of the blocking bug.

1. During playback, after modify the content in an editbox, the Save button is still grayed/disabled. A workaround for it is to click on other components like Check Box on the same page. e.g. Browser( ).Page( ).Frame( ).WebCheckBox( ).Click.
2. Before test web UI, open Internet Options -> Advanced, enable "Display a notification about every script error".
3. Bitmap checkpoint is too strict, hence too much false positive. Note that the bitmap screenshot may have an extra purple frame around it that needs to be removed. 
4. Optional step allows QTP to skip some failed steps, instead of terminating the test.
5. If the Debug menu is grayed/disabled, go to Windows Add/Remove Windows Components, and install Script Debugger.
6. Test Batch Runner is a useful tool in QTP program package to run test cases in batch.
7. To run external programs, use "SystemUtil.Run" function.
8. Able to run DOS commands.
9. Other programs can call the automation scripts created by QTP:

WScript /e: VBSCRIPT {path to the script}

          For more information, refer to QT Automation Object Model Reference -> Configure Settings for Running a Test.

    10.  To parameterize test data in script:

for n=1 to 3

......=database("{dataname}", "{datatable}")

datatable.SetNextRow

next

 

     11. Difference between GetTOProperty, GetTOProperties, and GetROProperty:

GetTOProperty:  property from test object description

GetTOProperties:  collection of properties and values used to identify object

GetROProperty:  current value of test object property

 

Our product had version 6.0, then 6.01, and now 6.02. And for each version, we have Windows release and Linux release. So when it comes to test cases management, it is kind of messy. 

Currently in our test case management system, we have 4 suites for 6.0 Windows, 6.0 Linux, 6.01 Windows, and 6.02 Windows  respectively. But every time when we execute the cases, say for 6.02 Linux version, we have to check out the cases from 6.0 Linux, 6.01 Windows, and 6.02 Windows.

It's not that simple as it seems because although the features between Linux version and Windows version are nearly the same, it has about 10% cases that should have different behavior, action, expected result, or even totally inapplicable; And old features from older versions have about 1~2% changes when they are upgraded to newer version. So when the QAs execute the test cases, they actually have another set of cases in their mind, where they assume applicable steps and expected results. However they never update the cases which actually are outdated for new version or different platform.

If I would do the project again, I would rather either

1. Put all the cases from different versions to only one suite, distinguishing them by adding a field for version number and platform type.

or

2. Seperate the test suites by versions and platforms, but copy the whole set of cases from older version to the new version suite, so that modification on the new version test cases would not have impact on the old version's.

and

3. For each version / platform test, switch the test owner of each module, so that the whole team can get the overall idea of the product and also enhance the test cases quality.

Last month, 2 interns leaved our team. About a couple of weeks before they leaved, they started to prepare for the materials and documents about the module they owned. And in the last week, they held a meeting with the successors to transfer all the information, and answered successors' questions about the modules during the whole week. After that, they leaved.

Everything sounds good, and it seems the successors got everything they needed to handle the modules. But when a real cycle of test came, a large amounts of practical questions popped up. Lots of the questions were about whether a certain behavior of the product was correct, how to set up the specific testing environment, what did the test case mean to do...

Shouldn't all these questions have been answered during the transfer of information, or their answers be found in the documents the interns prepared? Not until then did we realize that the transfer of information was not completed at all.

I think there are mainly 2 reasons for this:
1. The successors cannot really LEARN the modules and ask the correct questions until they practically test by themselves.
2. No standards/templates for the documents and materials they should prepare. And nobody review the doc they prepared either.

Based on the 2 reasons, I think first of all, we should start the transfer of information 2 weeks earlier, leaving enough time for the successors to finish a core cycle of test on the new modules. So that those practical questions (test cases, testing environment, expected results related) can turn up before the person leaves.

Secondly, we need a checklist. It's actually difficult to set up a template for the materials because test requirements vary. Some module may require special testing environment. Some may use specific testing tools. So maybe we can provide a checklist as detail as possible to make sure everything on the list are prepared. Here are some example items:
Feature spec, test plan, test cases (with clear description and expected result), environment set up guide, testing tools, automation testing scripts, known-issues, open bugs, trouble-shooting guide, other testing tips.

But even then, we should still have some responsible QAs review the docs and materials.

Lastly, I think it is better to prepare these documents during our working routines instead of doing it one month/week before one leaves. But this depends on every engineer themselves.

Sniffer Pro:

Example: Capture FTP transactions:

1) Define Filter -> Address:

    Set Address Type: IP, Mode: Include.

    Set Station 1. Dir. Station 2.

    -> Advanced:

    Check Availabe protocols -> IP -> TCP -> FTP (check other protocols as you need)

    Save.

2) Click Start button. And start the ftp transaction.

3) When Stop and Display button becomes activated, it means it has sniffed some packages. Click the button to stop and display the captured data.

4) Observe the Decode tab to find out the details.

5) Click Save button to save the capture.

6) Define Filter -> Data Pattern:

    Add Pattern: Choose the info you want to observe then click Set Data. eg. TCP: Source Port = 21.

UNC path: Universal Naming Conversion. Something like \\servername\sharename format.

DBCS / SBCS: Double-Byte Character Set (eg. Chinese, Japanese, Korean characters); Single-Byte Character Set (eg. English).

Email encodings:

Base64 (MIME): Transform binary sequence to ASCII sequence. sth. like SVY+PC9CT0RZPjwvSFRNTD4NCg==

QP (MIME): Quoted-Printable. Transform non-ASCII characters to ASCII characters. sth. like =ACK=A6=B3=A6=CA=AA=E1=AC=EE=A6=B3=A4=EB

UUENCODE: Transform binary data to ASCII. Start with "begin 666 {file_name}" and end with "end". 

BINHEX: An encoding method on Macintosh. Also transform to ASCII.

TNEF:  Transport Neutral Encapsulation Format (TNEF) is a Microsoft proprietary method for encoding Rich Text Format (RTF) messages sent across the Internet.

1. vim / vi  -- text file editing

* To open vim:

     run the following command:

     vim {file_name}

  If the file with {file_name} does not exist, it will create a new file with the name.

* To enter edit mode:

     press Insert or I

* To switch Insert / Replace:

     in edit mode press Insert

* To input long command (eg. wq):

     in command mode press :

* To exit edit mode or to input short command (eg. q):

     press Esc

* To delete a character:

     in command mode press X or Delete

* To scroll to the top of the file:

     in command mode press GG

* To scroll to the end of the file:

     in command mode press Shift+G

* To go down one page:

     in command mode press Ctrl+F

* To go up one page:

     in command mode press Ctrl+B

* To save & quit:

     :wq

* To quit without save:

     :q!

* To search a string:

     /{string}

* To find next:

     n

2. samba -- share folders

Quick started guide:

1) Open /etc/samba/smb.conf to do related configuration of the shared folder. Refer to the examples in the file.

2) Search "security=", and change its value to "share" (ie. security = share).

3) Save and close the file.

4) Add current user, run command:

     smbpasswd -a

5) Run:

     /etc/init.d/smb start

Done.

3. iptables -- linux firewall, packet filter

http://www.netfilter.org/documentation/index.html#documentation-howto

4. squid -- linux proxy

http://www.squid-cache.org/

1. Selenium

Open source.

Supports multiple platforms/browsers.

Does not support cross domain.

2. Quick Test Pro

Easy to insert steps in the middle of the original recorded steps.

Quickly recover from errors.

Reusable action.

Unintentional mouse movements during test case running would not fail the case.

Takes much time to add checkpoints. Bitmap checkpoint is not quite applicable.

3. Rational Robot

Supports multiple IDEs.

Supports multiple programming languages.

Not quite stable. Slow to recover from failure.

4. Mercury Winrunner

Also an automatic tool to record, examine, and simulate UI actions.

I'm testing a software on linux platform. Here are the commands that I use most:

/etc/init.d/XXX stop (or start) -- Stop (or start) service XXX. For example, /etc/init.d/postfix stop. On Redhat, the command can also be "service XXX stop (or start)".

df -- Check disk space.

du -abc -- Check the disk space occupation of the current directory.

find ./ -name "XXX" -- Find the file which filename contains string XXX in current folder. Wildcard is supported.

ifconfig -- Check the IP address of the system.

ipcs -- Check the shared memory allocated on a system.

netstat -nat -- Check TCP ports.

nslookup -- Check DNS.

PATH="{directory}":$PATH -- Add {directory} to the front of PATH environment variable. eg. PATH="`pwd`/bin":$PATH.

ps -ef | grep XXX -- Check the processes with the string XXX in name. Some more parameters give more details, eg. "ps aufxw | grep XXX", showing CPU time, and memory.

route -- Check the routes.

route add default gw {gateway ip} -- Add default gateway.

strings XXX | grep SSS -- Search string SSS in file XXX.

top -p {PID} [-p {PID}] -- Monitor the process with {PID}

vi (or vim) -- Enter edit mode of a text file.

zip -P {password} created_file_name {file name} -- Add password to a compressed file.