Regression Testing Algorithm
Regression Testing algorithm
When a large system has been developed and gone live, there are, inevitably, maintenance releases, how do you determine what to regression test? For example, on my current project we went live 12 months ago and we are putting out maintenance releases every 6 weeks. These maintenance releases are not purely live bug fixes, in fact only a small number of live bugs have been found. The majority of the maintenance release contain changes that the customer wants. These changes have been sitting around for some time as the customer wanted the core system to be live before implementing any changes.
Evidence during previous regression cycles has shown that a number of functions have displayed faults that were not in the previous versions of the code. This may be as a result of the changes breaking existing functions or bug fixes breaking functions or merged code causing faults or incorrect version control or other reasons. One of the difficulties with a large, complex, system is that a change in one part of the system can affect another part without anyone realising it. All the analysis in the world cannot stop this type of knock-on affect from happening.
This indicates that there is an ongoing need for regression testing of these functions.
It can be expected that a number of live bugs will be raised as a result of continued live running, so these have the potential to adversely impact existing functionality.
In order to perform a full regression test on the system it would take approximately 3 months for the entire team, I did say this is large and complex. Automation helps, but the system is such that large parts of it are either very difficult or impossible to automate.
Given the timescales for System Testing as part of a maintenance release it is not, therefore, practical to perform a full regression test, so the strategy is to perform what I call a rolling regression test.
The approach to regression testing at each maintenance release is as follows:
· Any function affected by a fixed bug or a change will be regression tested
· A subset of non-affected functions will be regression tested
· The Automated Regression Suite will be run at least once.
The “non-affected” functions are those that appear not to have been impacted by any change happening in the release. With over 400 functions in the system, each release should only affect a small number of these functions. (yeah, right!).
The algorithm for determining the subset of non-affected functions that will be regression tested is as follows (forgive the pseudo-code, it was the best way I could think of to get the message across):
BEGIN REGRESSION TEST ALGORITHM
Available regression test time = Total team time – (Time needed for Change testing + Time needed for Bug fix retesting)
SELECT
CASE been affected by Bug fix or Change
Add to Regression Candidate List 1
CASE been regression tested in the last released
No regression
CASE Business Critical Function
Add to Regression Candidate List 2
CASE ELSE
Add to Regression Candidate List 3
END-SELECT
END-FOR
FOR List = Regression Candidate List 1, Regression Candidate List 2, Regression Candidate List 3
FOR each function in List
IF time to regression test this function < Available regression test time
THEN
Available regression test time = Available regression test time – time to regression test this function
END-FOR
END-FOR
END REGRESSION TEST ALGORITHM
This algorithm will, therefore, cycle round the functions to ensure as broad a coverage as possible on regression testing functions. Over a number of maintenance releases, every single function will have been regression tested at least once.
