Software testing has gone through a few renaissances in the last twenty years
or so. From the dark ages of the waterfall process, there has emerged new
theories, schools and test tools. We even have online clubs and networks.
One noticeable difference between then and now is testing was traditionally
part of development. Now, most larger companies recognise the need for some form of testing team.
So donít you think itís strange when it comes to software testing, startups
still remain in the dark ages of developers testing their own code and then
skipping straight to a beta test? The rationale is that itís an easier, faster,
cheaper approach than employing a software tester.
I ashamed to say that until recently I bought into this misconception. Its
only when I started doing some research into beta testing, that I discovered the
amount of work and effort it took to. So Iím doing a Mythbuster session.
Myth 1: Beta Testing is easy
Beta Testing is hard work. Like any major task it needs planning and effort.
Successful Beta Testing takes lots of planning and lots of effort. Upfront
analysis into the number of Beta testers required and how they will be sourced.
How much time is necessary for the Beta Testing, how many releases will you
have (yes, you need more than one!). How will Beta testers sign up and will
their feedback need to be secure? How will you keep track of the bugs found, and
how much time will you need to support the effort? There is no point having
feedback from 200 Beta testers but not having the time to analyse the
information, let alone fix the problems. What tools will you need to keep track
of the Beta test? These are just some questions that will need to be answered
before you can begin your beta testing phase.
A couple developers talk about their experiences of Beta Testing..Joel
Spolsky - founder of FogCreek Software and
Myth 2: Beta Testing is Quick
The length of time beta testing takes depends on the complexity of the
software being tested and the number of releases you plan to have. Typical
estimates range from four to ten weeks depending it seems on your experience of
Beta Testing. One constant that remains, is that beta testing always takes
twice as long as you estimate. Thatís because one of the difficulties with Beta
Testing is your dependence on people who donít owe you anything and in that
sense are under no obligation to meet any of your deadlines.
Myth 3: Beta Testing is Free
The greatest myth of all. The man hours spent in planning, setting up,
monitoring makes beta testing a time consuming task. Time far better used by a
developer in turning the software into a great product. The end result is your
developer is distracted and performing tasks that many software testers can
perform quicker and cheaper. On top of that, testers know how to test software
really well, something you can never guarantee from a Beta tester.
Software Testers do it better
Software testing has really matured in recent years, and there are many ways
to provide quality testing that meet the unique demands that startups face.
Software testing does not necessarily equate with streams of documents and
restrictive process. You only have to look at techniques such as Rapid Software
Testing by James Bach and open source tools to know that there are many
alternatives to traditional software testing. There are so many good software
testers out there, available for hire on a freelance basis, it makes these age
old myths about software testing and startups, a bit passed their use by
Itís time for startups to do themselves a favour. Hire a software tester,
even for a couple of days. Itís worth every centÖ
Development houses have a right to expect a lot from a freelance tester.
Firstly, without the endless budget of some larger companies, they can ill afford time and money caused by improper scoping and testing.
Secondly, they have recognised the advantage of having an independent review of the product and that in its own right deserves to be appreciated.
In order to get the best return on investment from their tester, communication of priorities and expectations must be passed onto the tester. With knowledge, testers can focus their effort on what developer priorities instead of what they suspect is their priorities.
Iíve created a short questionnaire to clarify to developers what a tester needs. It contains some of these areas:
1) What do you as a developer value most? Consistency, Quality, Breadth of testing,
2) What specifically do you want tested in web testing?
3) What technology are you using?
4) What testing has already been performed?
5) Do you have any specs of any sort?
6) Is this new software, or updated software
7) What sort of feedback do you want? Defect reports, results?
Note: These questions have been created with web testing in mind, but can be changed for any type of testing
Hereís an example of what I use
Customer survey (pdf reader required)
Having this information upfront helps everyone because:
1) There is an agreed understanding of the scope of testing
2) Quotes can be validated through the questionnaire
3) More upfront information maximises your return on investment allowing focus on customer driven testing
4) It provides an insight into the software testing process
The above information is used to create tests in a spreadsheet which I use to track defects and results.
See example:Software Testing Template(pdf reader required)
With this document you have the benefit of evidence of testing which can be useful for contractual purposes. It also assists in future upgrades, streamlining the next round of testing.
I wrote a blog recently "the value of a software testing process" in which I questioned the traditional benefits of software test scripts based on the re-use and repeatability theory.
My approach to any rapid change software development is to make a some basic assumptions
1) The code is going to change quickly
2) The person who coded won't necessarily be there for the next release
3) The person who tested the last time, won't necessarily be there for the next release
4) Test documentation helps to provide structure and is excellent as a guide to ensure adequate coverage
5) Testers are able to think outside the square and aslo be able to fill in the gaps in a test
I use one spreadsheet (if possible) to track the following information;
a) test script number
b) test link (if available which is a reference to requirements etc)
c) test purpose - a clear concise description of what I am planning to test
d) test results - Pass or Fail
e) defect number - I assign a number of use the defect tracking system
f) test estimate - the time I anticipate it will take to test this functionality
I use the document to first scope out what I want to test, a quick
and dirty overview of what I am planning. I also use it to estimate how
long testing is going to take. This is helpful if I am ever asked to
substantiate my estimates.
Once all stakeholders are in agreement onto the scope, I create a
concise and descriptive purpose of each test. This is in essence my
test script. I use the same spreadsheet to enter results, defect
numbers and to calculate simple metrics such as the pass rate.
I have uploaded an example below;
Software Testing Template
Looking at the benefits of software testing from a business perspective can be quite a challenge if your a blue-hat, IT type of person as I am. To sell testing effectively though, its helpful to view testing from the perspective of the person who ultimately gets to make the decisions. So here goes!The way I see it, business is about Profit and Loss. So I've split up business as follows ;
1) Business want make money through sales
2) Business want cost cutting to improve the profit margin
Business want to sell things
If business is about selling products or services, how in business terms can software testing help sell a product? I've come up with the following possibilities;
1) Software testing discovers if critical functionality works. This is helpful to know when your trying to sell something (I'm assuming!)
2) Software testing makes sure that your product doesn't negatively affect interacting systems. I suspect this helps encourage repeat sales.
2) It provides tangible results which can be used to sell the product. For example, you can use your proven high performance as a selling point
3) It demonstrates delivery from a contractual perspective through acceptance testing
4) It gives confidence to those selling the product. There is added benefit in knowing the product your selling works.
5) Certification can provide business a selling point. If your system conforms to a technical standard, it may help your product to be perceived as reliable.
Business want to save money
How can testing help save a company money?
1) Early fault detection reduces the cost of fault detection. The earlier a defect is found, the less development rework and re-test is required, minimising its implementation cost. The Baziuk Study (1995) estimates the relative cost to repair a defect found in Operations to be between 470 - 880 times the amount found in the Requirements phase of the lifecycle*
2)It delivers efficiencies in the software development process through metrics such as root cause analysis . These detect possible areas of improvement for software development.
3) Software testing is the source of information such as defect reports, metrics and results that assist IT perform their roles efficiently. Project managers rely on metrics to report on progress, operations, on tangible results to extrapolate future hardware requirements and developers on defect reports to fix their code.
As is plainly obvious, I do not a heavy background in business, so I'd be most interested in comments from those who have!
And testers, how have you helped your company save money through testing?
*National Institute of Standards and Technology, May 2002
I don't know if any australians read this, but I'd like to invite you to a really great software testing club at:
Inside the club, you can start your own group which I have done for us Aussies as I think we could do with some representation.
Does a software testing process provide a best approach to testing?
Coming from a background in engineering I have always been a firm believer in the benefits of a sold and dependable testing process which includes test planning, design, building, execution and reporting. So, how come I find it hard to convince others of its merits? After much soul searching, I've come down to the conclusion and it's this. A rigid and formal testing process doesn't work in many of today's software projects. For example, some of the benefits of a software testing process are re-usability of documentation and repeatability of results
Re-useis cited as one of the benefits of a software testing process as it reduces long term cost both in resources and time. For example, test scripts once written can be used again in future releases reducing the upfront resources. But, in my experience, I've rarely used the same test script twice. This is because each time a new release is planned, its markedly different to the previous one. Add into the mix, new developers, new testers and soon the amount of flux necessitates a re-start of the test planning, design and building from first principles. I question the value of re-use when it ends up costing similar or more in time and resources.
RepeatabilityAnother benefit of a solid testing process is the repeatability of the results. Again though, if your code change is that significant, you will have to question the validity of the expected result. If resources are required to confirm or update the expected results, I'd have to argue time may be better spent in beginning again. This is especially true if the software tester is new to the project and requires analysis time to fully understand a product and its environment.
In summary I believe that where rapid change exists, you need an adaptable approach and is lighter on documentation. In industries with little change there is much benefit in a repeatable, re-usable approach, but where rapid change exists, I think spending lots of energy on paper documents which in the end will only get used once is a waste of time and cost.
Testing is essential, just that the process to perform testing must always be up for review and change too!
I run a small software test consulting firm, meaning it consist of me, myself and I. So I was very suprised to be contacted by an Indian software testing firm regarding partnership in offshore testing services.
Mildly flattered I suppose, but I asked myself, is this a direction I want to go in?
In my country, offshore software testing is regarded by many as the future of software testing. I'm assuming its a cost thing.
After some thought, I realised that really, its not why I started my own company and to be honest, not something I'm really interested in. I prefer to offer a small quality testing service and so I've probably answered my own question.
However, it does beg the question, apart from fiscally, are there other good reasons to consider such an option?
Some of you may be familiar with my previous blog on the work I am doing implementing a software test process. For those of you who haven't a bit of a background.
I'm doing some consuting work for a small finance company. They employ about 15 developers, but have no test team or testers. As an external consultant, I was asked to create a testing process almost a year ago. The creation of the software test process was the easy part. The hard part has been to try and get people to use it. The issues surrounding the implementation have been:
1) Lack of interest in testing
2) Lack of knowledge/experience (on testing)
3) Lack of time to allocate to testing
4) Lack of structure in the project team
Its made more difficulat because that they actually don't do that bad a job in testing the software at the moment. Where they really fall down is:
a) they can't proove they tested it well
b) their test management is pretty weak, which leads to poor resource usage and major delays in testing.
I am in the process of recruiting for a test analyst to help create/grow a test team. Hopefully that will help alleviate some of the issues.
I've started sitting with the projects and asking what are the issues in testing. I've then taken these issues to project managers and senior management who are working on fixing them. Specifically, the project managers need to be tracking the test effort as currently there is no test manager around to do the job
This week I want to suggest the following ideas;
a) I want to introduce some metrics which I can use to track our success.
b)I want to suggest quality awards, incentives, including use of process in the yearly review etc.
My two (aussie) cents for the day....
Kid in a candy store - 04:14, Sun 10 September 2006 - By amsinoz
I have the dream job at the moment .... I think. I've been asked to create a testing methodology for a company that wants to implement a software testing processe. Fantastic! I thought to myself, just what I've always wanted to do.
So I busily set about talking to people and finding out what they wanted and finally 3 months later we had this lovely document with templates which sits pride of place on my desk. And thats the problem, its about the only place where you can find it.
I did a couple of things, like creating a test office(they don't have a test team..yet) with representatives from each department. I also did some training with the test office and for each department on an individual basis. I then went away and left them to their own devices. Funnily enough, I got a call back 2 months later asking for more assistance!
So I came back and found the following problems
1) Some people just don't get it. Not everyone, but a large enough group don't seem to understand the point and to them its just another bit of paperwork to get through (or not as seems to be the case).
2) They don't seem to have time to implement it.One of the things I said at the start was that it was going to take more time to test in a structured way and to prepare for it, but the reality is that despite additional budgeting they still don't seem to have the time to perform the necessary tasks such as test preperation.
3) They feel its just too much work to introduce testing concepts such as creating a test plan, test scripts and reporting, so they don't do it. I haven't had much success on the defect tracking front either (excel spread sheets)
Its getting to the point where senior management are wanting results (no big suprise there!).
So I feel a bit like a kid in a sweet shop with all these different things to work on but I'm not too sure where to start.
This is what I have come up with:
1) Go to basics. Use minimum documentation (I'm thinking one spreadsheet) to take note of test script objectives, test log, defects and test report. Get them to use that and start taking some metrics down.
This will demonstrate that a) their is some form of process working and b) the metrics may prove vaulable in the future.
2) Get project managers to track actual effort against the project schedule to start tracking the test preparation, execution and reporting effort.
3) Go to the senior manager and ask him what he wants to get out of the process (what are his goals). Then work out the problems associated with the goals, work out some actions and try and benchmark against those goals.
Thats about it so far, I'd value anyones comments on this one!
Selling Testing - 02:38, Tue 5 September 2006 - By amsinoz
I've been working on creating a testing methodology for a financial institution who are keen improve money without a) spending anything and b) adding any extra effort. Its been up to be to sell the benefits of testing to them. Its been interesting for me, as my background is engineering and so to me you just do testing because thats the way things are done. Unfortunately business don't see it that way. I read this great article by James Bullock on "Calculating the value of testing". It focused on how you really have to sell testing to business. I suppose it might be obvious to most, but to simple me it was a 'light bulb' moment and since then I have tried to really think of how non-technical people view testing. I haven't yet found out the perfect selling pitch but look forward to the day when I do.
I dont think you can just strut well used 'benefits' of testing such as better quality, lower cost (eventually). I think to sell testing you have to strike a chord - hit a nerve, whatever, to really catch your clients attention. To get that, you have to be willing not to just sell, but to really listen.
My two cents worth for the day....