Software testing has gone through a few renaissances in the last twenty years or so. From the dark ages of the waterfall process, there has emerged new theories, schools and test tools. We even have online clubs and networks.

One noticeable difference between then and now is  testing was traditionally part of development. Now, most larger companies recognise the need for some form of testing team.

So don’t you think it’s strange when it comes to software testing, startups still remain in the dark ages of developers testing their own code and then skipping straight to a beta test? The rationale is that it’s an easier, faster, cheaper approach than employing a software tester.

I ashamed to say that until recently I bought into this misconception. Its only when I started doing some research into beta testing, that I discovered the amount of work and effort it took to. So I’m doing a Mythbuster session.

Myth 1: Beta Testing is easy

Beta Testing is hard work. Like any major task it needs planning and effort. Successful Beta Testing takes lots of planning and lots of effort. Upfront analysis into the number of Beta testers required and how they will be sourced.  How much time is necessary for the Beta Testing,  how many releases will you have (yes, you need more than one!).  How will Beta testers sign up and will their feedback need to be secure? How will you keep track of the bugs found, and how much time will you need to support the effort?  There is no point having feedback from 200 Beta testers but not having the time to analyse the information, let alone fix the problems. What tools will you need to keep track of the Beta test?  These are just some questions that will need to be answered before you can begin your beta testing phase.

A couple developers talk about their experiences of Beta Testing..Joel Spolsky - founder of FogCreek Software and SyneRyder Journal

Myth 2: Beta Testing is Quick

The length of time beta testing takes depends on the complexity of the software being tested and the number of releases you plan to have. Typical estimates range from four to ten weeks depending it seems on your experience of Beta Testing.  One constant that remains, is that beta testing always takes twice as long as you estimate. That’s because one of the difficulties with Beta Testing is your dependence on people who don’t owe you anything and in that sense are under no obligation to meet any of your deadlines.

Myth 3: Beta Testing is Free

The greatest myth of all.   The man hours spent in planning, setting up, monitoring makes beta testing a time consuming task. Time far better used by a developer in turning the software into a great product. The end result is your developer is distracted and performing tasks that many software testers can perform quicker and cheaper. On top of that, testers know how to test software really well, something you can never guarantee from a Beta tester.

Software Testers do it better

Software testing has really matured in recent years, and there are many ways to provide quality testing that meet the unique demands that startups face. Software testing does not necessarily equate with streams of documents and restrictive process. You only have to look at techniques such as Rapid Software Testing by James Bach and open source tools to know that there are many alternatives to traditional software testing. There are so many good software testers out there, available for hire on a freelance basis, it makes these age old myths about software testing and startups, a bit passed their use by date.

It’s time for startups to do themselves a favour. Hire a software tester, even for a couple of days. It’s worth every cent…

OOP concepts:

• encapsulation - Conceals the functional details of a class*.
• modularity - Simple parts connected by clean interface.
• polymorphism** - Different data types respond to methods of same name
• inheritance - Subclass, specialized version of class.

*Class - abstract characteristics of a thing (object)

  Object - pattern of a class

  Instance - actual object created at runtime

  Multiple inheritance - one subclass has more than one superclass

 

**Polymorphism:

• Override - Child-class overwrites parent-class function (NOT private)
• Overload - Functions with same name, but different parameters (type / number / order) (NOT private in parent)

If a function would have different behavior in subclasses, it should be virtual.

If a superclass (base class) has a virtual function, it should be overridden.

OOP languages: C++, Java, Python, Perl ...

	C++/Java	Python	Perl
integers	primitive native types	objects (of int class)	treated as double float
self pointer	this	self	(any)
constructor	classname()	__init__	NEW
destructor	~classname()	__del__	DESTROY
class members	public, private, protected	all public	all public
methods**	public, private, protected	all virtual	all virtual
private variable	private xxx;	__privatevar	all public*
pass by value/reference	C passes by value; Java passes references to objects by value.	Passes references to objects by value.***	Pass by reference - hard / symbolic

* Perl coding convention: use _ or __ (double underscore) before variable to indicate private. But not enforced.

** Subroutine:

• function: "return xx;"
• procedure: no return. Sometimes synonymous with function, eg. in C.
• method: OOP concept, part of objects

*** Python 2 kinds of data structure:

Immutable: string, tuple

Mutable: list, dictionary

 

SMTP:

Commands:

one session:

HELO

MAIL FROM: <...>

RCPT TO: <...>

(RCPT TO: <...>)

DATA

...

.

QUIT

Other commands: RSET, NOOP

Response: 1xx, 2xx, 3xx, 4xx, 5xx

POP3:

Commands: USER, PASS, STAT, LIST, RETR, DELE, NOOP, RSET, QUIT, TOP msg n

Response: +OK, -ERR

IMAP vs. POP3:

IMAP:

• Client does not download the whole mail, only downloads the mail header.
• Client can directly manipulate the mails on server.
• User can maintain the mail directory on server.
• User can retrieve specific part of a mail, eg. text.  

FTP:

Commands: USER, PASS, RETR, TYPE (A/E/I/L), PASV, STOR, DELE, LIST, PWD, NOOP, QUIT, CWD

Response: 1xx, 2xx, 3xx, 4xx, 5xx

HTTP:

Simple:  GET URI HTTP/version

Full:

(General-Header                  - Date / Pragma

 Request-Header                  - From / If-Modified-Since / Referer / User-Agent

 Entity-Header)                     - Content-Encoding / Content-Length / Content-type /

                                               Expires / Last-Modified

 CRLF

[Entity-Body]                       *OCTET

Response:

1xx     informational

2xx     successful

3xx     redirection

4xx     client error

5xx     server error

		Servers	Clients
HTTP	Windows	IIS; Apache	IE; FireFox; Netscape; Mozilla
	Linux	Apache	FireFox; Netscape; Mozilla
FTP	Windows	IIS; ServU	WSFTP; CuteFTP; FFFTP; FlashFTP; SmartFTP
	Linux	proftpd	(command line)
Mail	Windows	IIS; MS Exchange; KerioMailServer	MS Outlook; Outlook Express; Foxmail; Eudora
	Linux	Postfix	(command line)
Proxy	Windows	CCProxy	/
	Linux	squid	/

Generic Test Plan:

1. Reference Documents (Requirements)

*2. Objective (Prioritize functionality / usability / reliability /security / stability / sustainability / portability / efficiency ...)

*3. Scope / Test types (and pass criteria)[1]

4. Test environment (software / hardware / network / other facilities)

5. Test tools

6. Human resources (team member vs. responsibilities)

*7. Schedule

8. Bug priority / severity definition

*9. Alpha / Beta / release criteria

10. Risk analysis (mitigation plan)

* I consider must-have.

[1] Test types:

• Functional test - normal, boundary, force error
• System test - compatibility, conformance (standard), integration, volume / load (performance / stress), I18N / L10N, security, stability test
• Coverage test

Performance Test Plan:

1. Objectvie /Goals (requirement, pass / fail criteria)

2. Navigations (business process, prioritize)

3. Scenarios (min / max / ave. load)

4. Environment

5. Monitors (measurements)

6. Schedule

7. Resources

8. Risks

UI test tools:

     QuickTest Pro (QTP)

     IE7 Readiness Toolkit -- for IE7 compatibility test

     Sprajax -- assesses the security of AJAX-enabled web applications

     Rational Robot

Installation test tools:

     InCtrl5 (windows) -- monitors the changes made to the system

     SciTe for AutoIt (windows) -- auto-installation

Resource leak test tools:

     Valgrind (linux) -- memory leak, etc.

     Rational Purify (windows) -- memory leak

Stress test tools:

    Canned HEAT (CH3) (windows)

    stress-0.18.9 (linux) -- stress on cpu, memory, disk space, etc. 

Test Coverage tools:

     Bullseye (windows / linux)

     Cobertura -- based on jcoverage. Used for java program.

     Rational PureCoverage (windows)

Performance test tools:

     Jmeter (windows / linux) -- open source.

     LoadRunner (windows)

Security test tools:

     IDS test tools -- IdsTest (linux), Hping (linux), netwox (windows)

     Security QA toolbar -- SQL injection, LDAP injection, XPATH injection...

Fuzzing test tools:

     JBroFuzz

     Webscarab

     Suru

     Sulley

Web app test tools:

     WebInject

     curl (linux)

     MS Web Application Stress Tool

     Security QA toolbar

System monitors:

     CurrPorts (windows) -- list of opened ports and the processes.

     Informer (windows) -- reports system status.

     TCPView (windows) -- list of running processes and more details.

Data capturers:

     HttpWatch

     Sniffer pro (WireShark)

Test cases optimizing tools:

     allpairs -- constructs a reasonably small set of test cases that include all pairings of each value of each of a set of parameters. Supports diagnostic output.

     pict -- similar to allpairs. Supports sophisticated options.

Remote control:

     PsTools (windows) -- PsExec: execute programs on remote systems; PsFile: shows / closes a list of files on a system that are opened remotely; etc.

     Remote Administrator (windows)

     Remote Desktop (windows)

     Putty (linux), Pietty (linux, cn / jp locale) 

     SSH

Test Automation Framework:

     STAF

Installshield:

     PackageForTheWeb -- e.g. pack a hotfix

     IExpress

Project management:

     WBS + PERT + MS Project

     WBS -- for constructing plan and schedule

     PERT -- for estimating most possible ending date

     Project -- for making project plan, tracing and controling progress

Virtual Machine:

     VMWare (VMWare)

     Virtual PC (Microsoft)

     VirtualBox (Sun)

Source code control:

     SVN (Subversion)

 It's been 3 months now since I joined a new company. I’m still getting used to it. I mean, there are so many differences between working as a QA in an SMB like my previous company, and a huge enterprise like my current employer.

1. New hire guidance

In my previous company, which is much smaller, they are able to provide a clear list of “What to do” for new hire, like whom you should contact if you are in some situation, or where you can seek help when you are in some another situation.

My current employer provided a first day orientation, which propagandized company culture, and gave us a website, which surely contains all the information we need yet it’s too huge. My direct manager did most of the other orientation job afterwards, but some information he provided was either outdated or conflicted with what HR said later. I was confused sometimes, trying hard to figure them out myself.

2. Organization

My previous company is simpler, from director, to XXX managers, to developers and QA engineers, straightforward, and people know each other. One RD team works on one project, developers program and QAs test.

My current employer is much more complicated. Different kinds of teams, different kinds of roles, even QA has several categories, focusing on different kinds of testing on the same project. I always wonder if we will miss anything in the cracks between these QA teams. 

3. Process

The previous company has an SQA team that supervises the product development cycle to make sure it adheres to process. They involve in requirement review, design review, plan review, and then after release, they analyze the testing data and do postmortem.

Now we do not have this kind of force in my company. We are working on many small releases of old products, and have short time span and not very strict process. No hard test coverage requirement, how many tests to do depend on QA’s experience and developer’s suggestion. Bug priority at 3 or lower are sometimes ignored, and product goes on releasing. Usability, UI, comprehensive messages are not a bit in consideration. Nobody cares about that because there are so many projects on one’s hand at the same time.

4. Culture

In my previous small company, employees’ race, age, experience, and even character are close. We worked together, played games together, and even rent an apartment and lived together. It’s pretty simple, and people like each other.

In a big old company like the one I’m in today, we have so many colleagues from different countries, in different age, and have different experiences. People do not hang out a lot after work. They don’t see each other even working on the same floor. They prefer conference call than physical show up. In a team meeting, if one’s not aggressive enough, he would have no chance to chip in, because there always are talkative members out there that really show they are involved.

The good thing is, in a big company, you always have something new to learn, either from projects, or from the coworkers.

The test result may not be authority, but using the same criterion to test every time, it is convenient enough.

WebWait is a tool for testing specified website load time.

If a site loads slowly, user will soon go to other alternative sites. To prevent that from happening, sites need to do accurate check on speed before release.

It is good to use this tool to compare the results from your own site and other sites.

Further more, you can set multiple tests interval. e.g. test 5 times every 60 seconds, and it will calculate the average load time automatically (to avoid accidental test result).

To make a comfortable site, it is a good idea to make use of this tool.

I think this really simple task management tool would be good for Agile development.
Scrumy does not require member login, and supports simply drag & drop tasks.
It's very easy to access, just add the project name after Scrumy's domain name and that's it.
Following that is to add stories and tasks. Drag & drop tasks to move between To Do, In Progress, Verify, and Done.
It is clear to anybody what the tasks are and where they are. Different owners have different colors on their task sheets. It's amazing.
Since there's no member login requirement, security is quite loose. But it is pretty convenient for quick projects within a team.

Dead-Links is a simple tool that helps you to check whether the links to the page under the URL you specified are broken.

If some links on your website are broken, it may not only embarrass you, but also lose some users or customers.

There are similar tools already, but this one has an explicit domain and simple interface, it's convenient to use.

Although it takes some time to check, it's not bad to do periodic check on your own blog or site.

This is a good concept. Improvements on the other screen capture tools I used.

Bug Shooting is basically screen capture software. The selling point is that it can add comments and markings onto the screenshot, and send to the bug tracking system immediately.

It is true that when reporting a bug, attaching a commented screenshot is helpful, but sometimes the process is troublesome. So this tool manages the whole process and makes bug submitting easier.

Here's a short tutorial for the tool.

From "High Performance Web Sites", Steve Souders.

1. Make fewer HTTP requests
2. Use a CDN
3. Add an Expires header
4. Gzip components
5. Put CSS at the top
6. Move JS to the bottom
7. Avoid CSS expressions
8. Make JS & CSS external
9. Reduce DNS lookups
10. Minify JS
11. Avoid redirects
12. Remove duplicate scripts
13. Turn off ETags
14. Make Ajax cachable and small

1. How to migrate your codes to support Unicode:

1) Add: #include 〈tchar.h〉

2) char -> TCHAR

3) "text string" -> TEXT("text string")

4) Use CharNext & CharPrev instead of ++ & --

2. In MS Excel, how to freeze the row/column header, or split the views of a sheet:

1) Window -> Split

2) Window -> Freeze Panes

3. In MS Excel, how to add a filter button next to the row header:

1) Highlight the whole header row.

2) Data -> Filter -> AutoFilter

4. In Exchange server, how to forward mail to another mail server:

1) Open Exchange System Manager.

2) Administrator Groups -> First Administrator Group -> Routing Groups -> First Routing Group -> Connectors -> New -> Routing Group Connector

5. How to encrypt a folder in Windows system:

The folder Properties -> General -> Advanced -> Encrypt contents to ...

6. How to use FTP proxy while browsing in IE:

1) Internet Options -> Advanced: Disable folder view for FTP.

2) Set FTP proxy in Internet Options -> Connection.

7. In Outlook, how to insert a mark before every comment you make:

Tools -> Options -> E-mail Options -> Mark my comments with

8. In Outlook, how to add voting buttons to your mail:

New a message. Open Options. Find Use voting buttons.

9. How to create a new LAN with VM:

1) VM settings -> Hardware -> Ethernet: select Host-only or NAT for Network Connection.

2) Set VM Default Gateway to the host VM NAC IP.

10. How to install perl module in Windows:

>  If the downloaded file is .tar.gz. For example, you can get these types of files from http://search.cpan.org/

   1) Extract the archive.

   2) Read the Readme file for how to install the module. Usually it includes the following steps:

    perl MakeFile.pl

    nmake

    nmake test

    nmake install

Note: Download and install nmake first.

>  If the downloaded file is .ppd / .tgz. For example, you can get these types of files from http://ppm.activestate.com/PPMPackages/zips 

   1) Open command window, and run:

    ppm install C:＼pathtomodule.ppd

11. In C/S environment, how does client renew server ip - hostname info:

Run command: nbtstat -R

Example, server has changed IP. After running this command, client can successfully ping server hostname again. 

12. How to enable/disable UAC in VISTA:

1. Launch Local Security Policy. (Tips: type "sec" in Run editbox, it will automatically list the program names.)

2. Open Security Settings -> Local Policies -> Security Options -> UAC.

13. How to reboot system by command:

shutdown -r -t 0

14. How to install Active Directory on Windows Server:

Run command: dcpromo

15. How to configure SQL 2005 to accept SQL Authentication:

1) Right-click on the server node and select 'Properties'.
2) Select 'Security' from the left menu under 'Select a page'.  
3) Under 'Server Authentication', select the 'SQL Server and Windows Authentication mode option'.
4) Click 'OK' to close the dialog.
5) Right click on the server node and choose 'Restart' for the changes to take affect.
Enable SQL Server login

1) In the server node expand 'Security' and 'Logins'.
2) Right click on the login name and select 'Properties'.
3) Enter a password and confirm the password for the login. 
4) Select 'Status' from the left menu under 'Select a page'.  
5) Set the 'Login' option to 'Enabled'.
6) Click 'OK' to close the dialog.

Cited from http://kbase.gfi.com/showarticle.asp?id=KBID002804

16. How to hide partition:

1) Run Regedit.

2) Under [HKEY_CURRENT_USER]-->[Software]-->[Microsoft]-->[Windows]-->[CurrentVersion]-->[Policies]-->[Explorer], add a DWORD value [NoDrives], and set the decimal value as follows: set [1] to hide partition A, set [2] (A double) to hide partition B, set [4] (B double) to hide partition C, etc. Set [67108863] to hide all.

3) Under [HKEY_LOCAL_MACHINE]-->[Software]-->[Microsoft]-->[Windows]-->[CurrentVersion]-->[Policies]-->[Explorer], add a DWORD value [NoDrives], and set the decimal value as follows: set [1] to hide partition A, set [2] (A double) to hide partition B, set [4] (B double) to hide partition C, etc. Set [67108863] to hide all.

17. How to change multiple OS boot up priority in Windows:

C:＼boot.ini

Sites and services put more and more attention on usability.
When a real user is trying the site, she talks about what she feels about it. This information gives great inspiration on how to improve the site or service.
That is what Silverback is for. Although it's for Mac OS only and sells about $50,
I think this kind of software is necessary.
The software is able to record tester's operation, facial expression, and talk, and makes a Quicktime video.
I think the tool is simple but amazing for usability test.

However I have not tried it yet, since I don't have Mac OS.

FITNESSE and NOSE are both test automation framework. FITNESSE is usually described as a functional and integration test tool. NOSE is usually said to be a unit test tool.

I evaluated both of the tools before we select one for our project testing. 

FITNESSE has a wiki UI, and uses tables to manage test cases and results.

NOSE does not have web UI, but all cases execution summary is displayed in command console.

Here are some more comparisons:

                                  |     FITNESSE    |    NOSE
1. Checkpoint/assert   | in the table         |   in the codes
2. File/Function          |                           |
   naming convention   |   camel               |    test_XXX
3. Case executed       |                           |
   sequence                 | as in the table     |  prioritized by case name
4. Result storage        |  in the table         |  in command console/has API

Conclusion:
FITNESSE is fit for data driven test cases, which need a table of data for similar case.
NOSE is fit for module test, with no UI, and flexible to use its APIs.

In some meetings, I found most of the team members were absentminded. I summarize the common ground of these meetings.

1. The meeting request is not sent in advance.
If meeting request is not sent at least 1 day before the meeting, attendances may have their own schedule for the day, and may not be comfortable to adjust it because of the sudden meeting, nor have time to get prepared for it.

2. The meeting does not begin on time.
If the meeting environment is not well prepared in advance, or the meeting is not started on time, attendances may be discouraged and have impact on focus during meeting.

3. The meeting host is the presenter.
Hence nobody control the presentation time and the content may be unconstrained.

4. The meeting does not end on time.
If the meeting time is out of control, attendances will probably feel loathing and start thinking about their own stuff.

5. The meeting background light is too dim.
If only the screen in front of the meeting room is flashing during the whole meeting, attendances may feel sleepy. So proper background light is very necessary.

We held a workshop to determine the 1st stage feature list (SFL). Since our product manager is not at the same site as the development team, we decided to work out the SFL within the development team first, then send out to product manager for review.

What we did in the SFL workshop is like this:

The attendees of the workshop include developers, QAs, development manager, QA manager, project manager, and UI engineers.

First of all, we filter the categories of features in the full feature list (FFL) that would not be implemented in the 1st stage.

For the remaining categories, all of the attendees have to score each single feature by its implement priority - score 3 for high, 2 for medium, and 1 for low. At the same time anyone can give an explanation for why he scored that point. Each role stands from his point of view to decide the priority. And as a QA, we tried to figure out the simplest story that an end user can try out our service at once.

After going through all the features in FFL, we got a total priority score for each feature. Then we sorted them descendingly, and review the top 20 highest priority features to check whether they make sense.

Then we sent the list of features with priority order decided by development team to our product manager. To our surprise, his feedback was totally different from our feature list. The product manager focused more on the infrastructure and high availability stuff. So we combined his feedback and adjusted our SFL for the 1st stage. We focused on a more scalable structure and high availability, and only kept the simplest feature that user could use.

After decided the SFL, we kicked off the 1st iteration by starting estimating the 1st stage schedule. Here is how we did this:

The attendees are the same as SFL workshop except UI engineers because they were shared resources and would not be in critical path.

First of all, we separated 3 phases for each feature implementation - design / test environment preparation, coding and unit test / automation test coding, bug fixing / testing. You can see that each phase involves task for developer and QA. So we estimated the working hours for each phase of each feature in the SFL respectively by developer and QA. And we found the critical path. The sizing and time estimation was mainly based on experience, kind of intuitive and conservative though.
After that, we also assigned owners to each task, trying to balance all engineers' effort.
Of course, finally we added buffer time (about 20%) and the time for other stuffs such as full cycle test and regression test. 

And that's how we planned our 1st iteration.

Thanks for Michael's comment. And now we've worked out a draft of working model for our agile development. We will try it out, and refine it during practice.

In the graph above, FFL is for Full Feature List, SFL is for Stage Feature List, and TDD is for Test Driven Design.

Two inputs for FFL: new feature requirements from product owner, and user feedbacks.

For each iteration of development select a set of stage feature list.

Each iteration adopts TDD mentality.

After an iteration, it may or may not deliver a beta release. But no matter deliver it or not, the program is shippable after each iteration.

Currently we are entering the first iteration, meaning we will decide a stage feature list for the first stage shortly. We believe the first stage, or say iteration, would cost longer time, because we have to consider the whole architecture and scalability.

I may write another blog about how we prioritize and decide the first stage feature list.

For the job:

1. Learn to analyse the essential of a problem, or the root cause of an issue.

2. Do things with a plan. Evaluate the value of effort before doing it. (eg. automation)

3. Communicate to get clear about counterpart's expectation. Avoid over effort.

4. Work with the team, as a team. Take care about the gray areas where task owner is not clear.

Can I become more senior this year?

To avoid becoming more and more conservative in releasing our product, the executives ordered that we introduced Agile mentality, and made our software development AGILE!

Well, before that, we do not have any clue about what Agile is, neither we have a successful agile pilot team. So we, the development group, started whitehanded.

Does it mean shorter schedule, no documentation, less test...?

After a couple of weeks of study and discussion, we found some difficulties in implementing Agile process (maybe it's not called a process for Agile) from our engineers' point of view. And the top 3 are:

1. Instant face to face communication among all stakeholders are not possible.

Because we are a transnational company, and the stakehoders, i.e. marketing team, development team, sales, tech writers, decision making managers, etc. are locating through out the world in different time zones.

2. Lack of senior, experienced engineers.

All developers and QAs in our team have only 2~5 years software development experiences. And nobody have ever involved in an agile team. 

3. Working model is not decided yet a new cycle of product development is on the road.

We have started our new version of product development, but how to work agile is still in discussion.

I will follow up what happen on our agile development in my coming blogs.

1) Program files and folders.

2) Program related registry entries (regedit).

3) Related processes (task manager).

4) Related services (services.msc).

5) Shortcuts on desktop, Start menu, Quick Launch, and any other places in which may have shortcuts added during installation.

6) Program name in Add or Remove Programs in Control Panel.
7) Items added in OS settings. For example,
(a) Windows firewall exception list, including program name and service name in advanced settings.
(b) Services added in Local Area Connection Properties.

The best thing to do is to ask the developer for a list for What is Done during installation, then check one by one after uninstallation for what has been left.