One thing I have been interested in over the years is the question of determing when testing is done. When enough testing has been completed and the product is ready to be shipped. In the past I have used completion metrics such as Defect Resolution Rate (how fast are you fixing things and when are you fixing more than your finding) and Test Execution Status to aid in this task. Test Execution Status can be the amount of coverage of the requirements (test cases covering requirements) and their pass/fail rate along with execution rate (tests planned and executed) of the tests. This show the traditional growth curves when plotted. It give you an idea of how much you have done. Which helps to determine where you are at in the cycle.
An interesting thing I have read about in relation to the Agile methods (SCRUM in particular) is the 'Burn Down Rate' which is used to show how much is left to do for 'stories' (requirments/use cases) over a specified time period. It is a downward sloping curve that can used to say you have so much left until the cycle is complete. Considering this is more in line with what we get asked (how much testing is left to do, and when will it be done by) by management it may be a more effective way of also showing how much testing is left. Instead of stories you plug in the test cases for those stories, and you plot the trend of how fast you are 'burning down' the list of things to do. This way you could say to management we have X percent of test cases left to execute and we look to have them done by Y date. This is what they want to hear.
I have not done this on a project yet, but am looking forward to trying it out. After reading this if you have implemented it please respond back with your experiences. If not, then maybe this will inspire you to change your perception of test execution status reporting & monitoring, and to see if you can 'burn down' some of your testing.
This method of determining how far through testing you have got is not an Agile-invented method. I learnt this about 15 years ago as part of a large project that was being delivered in stages. It went through design, development, test phases and during each test phase we calculated how much testing was left to do until the cycle was complete.
Couple that with some of your more "traditional" measures and you can present a good picture to give your managers.
I have produced some interesting analysis of the burn rate vs bug raising vs bug fixing that can tell you quite a bit about the state of the software and the state of the testing.
The interesting thing about coverage is that I try to measure it in two stages. First when the test design is complete (test cases and/or initial Exploratory sessions) and secondly as the testing is performed. I like to know ahead of the testing how much coverage we expect to achieve and when. This enables you to plot coverage against tests performed and perceive a burn rate on coverage as well as tests executed.
As you can see from the WebLog title I am a bit sarcastic and cynical about this thing we call Software Testing. Over my years of experience in Software Development and Testing I have seen some very very Dilbert things happen.
Hopefully this Blog will be a good place for you to learn from some of the things I have experienced and allow you to be more effective in your efforts in Software Testing.
• June 5, 2006 - Burn it down
One thing I have been interested in over the years is the question of determing when testing is done. When enough testing has been completed and the product is ready to be shipped. In the past I have used completion metrics such as Defect Resolution Rate (how fast are you fixing things and when are you fixing more than your finding) and Test Execution Status to aid in this task. Test Execution Status can be the amount of coverage of the requirements (test cases covering requirements) and their pass/fail rate along with execution rate (tests planned and executed) of the tests. This show the traditional growth curves when plotted. It give you an idea of how much you have done. Which helps to determine where you are at in the cycle.
An interesting thing I have read about in relation to the Agile methods (SCRUM in particular) is the 'Burn Down Rate' which is used to show how much is left to do for 'stories' (requirments/use cases) over a specified time period. It is a downward sloping curve that can used to say you have so much left until the cycle is complete. Considering this is more in line with what we get asked (how much testing is left to do, and when will it be done by) by management it may be a more effective way of also showing how much testing is left. Instead of stories you plug in the test cases for those stories, and you plot the trend of how fast you are 'burning down' the list of things to do. This way you could say to management we have X percent of test cases left to execute and we look to have them done by Y date. This is what they want to hear.
I have not done this on a project yet, but am looking forward to trying it out. After reading this if you have implemented it please respond back with your experiences. If not, then maybe this will inspire you to change your perception of test execution status reporting & monitoring, and to see if you can 'burn down' some of your testing.
• June 6, 2006 - It is not just Agile methods!
Nice item.
This method of determining how far through testing you have got is not an Agile-invented method. I learnt this about 15 years ago as part of a large project that was being delivered in stages. It went through design, development, test phases and during each test phase we calculated how much testing was left to do until the cycle was complete.
Couple that with some of your more "traditional" measures and you can present a good picture to give your managers.
I have produced some interesting analysis of the burn rate vs bug raising vs bug fixing that can tell you quite a bit about the state of the software and the state of the testing.
The interesting thing about coverage is that I try to measure it in two stages. First when the test design is complete (test cases and/or initial Exploratory sessions) and secondly as the testing is performed. I like to know ahead of the testing how much coverage we expect to achieve and when. This enables you to plot coverage against tests performed and perceive a burn rate on coverage as well as tests executed.