Rainbow Testing

Count down to release – is your testing plan ready?

Don’t let implementation catch you by surprise. The testing needs of the release effort are somewhat different than your mid-cycle activities. But don’t panic, either, this is well within your range. First let’s look at where you are in relation to the plan.

Release metrics in the plan – check the obvious.

If you’re lucky you’re doing this research while in the test planning stage. You can be quite optimistic about your readiness for implementation when that stage arrives. If you’re somewhat less lucky you inherited the test plan and it has a bare bones release plan included. You’ll want to take a look at that.

If you’re in a panic it’s probably because your release date is looming and there is no implementation plan. Take a deep breath and read through this. There are no guarantees, but you may find some helpful and sane suggestions here.

The test plan will generally contain the release criteria, and will usually indicate the conditions related to test case status and open defects that must be met prior to release of the code. It may say something like this:

“No critical or high priority defects are open. Any medium or low-priority open defects are signed off as acceptable risks All high priority test cases have passed. User Acceptance Testing has been completed and the user group representative has signed off.”

The standard plan insert is a good start. But no need to stop there. Schedule the meeting to review the project release plan and begin a more thorough review. Schedule first? you ask. Yes, ready or not the review is a must have. Get it on the calendar as soon as you reasonably can.

Defect review – the next step

Remember that the standard is just a start. You need to know where you’re really at. Start with defects. If you’ve been tracking them consistently you’ll have a lot of information there. Start by  reviewing the current defect reporting and trendlines and looking for a final spike assoc with regression a.k.a the “bug bounce”

Is the trend otherwise downward? That’s important. In particular your mid and high level defects should be tapering off. If there’s an upward trend in trivial defects this may indicate that your testers have the time to get nit picky – a good sign for release.

Take a moment and analyze defect clustering – make a note of hot spots – we’ll refer to this note later.

Now you’ve got some “known knowns” Here are some activities that you might put on your list:

If there is no bug bounce, check on regression

If the trend is upward, or flat but high, check on over all test progress.

If there are hot spots, check on this area of the test plan for coverage and status

If there are no high priority/critical defects, there was a bug bounce and the trendline is downward, you can feel generally reassured that you’re ready for the release meeting that you scheduled. But don’t stop there, go searching for unknown knowns,

Look for the gaps: review with team members

Communication is paramount, you’re in the final minutes of the final game of the championship series, everyone on the field should be aware of everyone else and aware of the position of the ball at every moment. This is what you need to know:

are all defects accounted for

all statuses correct and up to date

How close are you to the project release metrics?

If you are at the release metrics – it’s time to project the release date, schedule the release activities and make a list of the items that block you from release that might not have been on the release plan.

If you are not at release metrics – déjà vu!  it’s time to project the release date, schedule the release activities and make a list of the items that block you from release that might not have been on the release plan. However in this case you may need to raise the level of awareness on blocking issues.

Set up for awareness.

If your defect tracking system has this feature, set up automatic notice of new defects – possibly by severity or priority.

Face the music on lingering outstanding issues; make explicit plans for the defects that will not be fixed for the release. You may wish to put them in the release notes and/or convert to them to enhancement requests for the next upgrade.

Testing review

Defects aren’t your only metric, look at the test plan and execution metrics. Test reporting can be less straightforward than defect reporting. How you report your testing results can change the best approach for summary reporting.

A tale of two test groups

One test group selected specific test cases from the plan into test sets and tested the same cases over and over again directly linking defects to failed test cases.

They looked for progress over time from consistent test sets and cases.

The second team selected at random from planned functional group test cases using different cases within equivalence classes, looking for progress over time for different test sets and cases.

Different methods, one project, one report, two approaches.

Reporting for the first group could be done at regular intervals reporting the current status of their test set.

Reporting for the second group needed to be done out of the test plan, with explanations for failed test cases that were not run again.

The reporting could be done in a single Excel work book with separate tabs and a central report summary.

What are the important questions when it comes to taking the pulse of your testing status. Here are some suggestions

What is the status of your current testing?

Has the UAT been completed?

Are all the results logged?

When reviewing the planned testing you want to pick out all test cases that have failed and not passed again as well as all test cases not run. When you schedule the final smoke, or exit testing – can you include the stones unturned? Try to include the high priority tests that you just picked out in your review to change the reporting status for them.

In the end you need to certify that any test cases not run cover functionality that has been covered by other tests or are signed off as acceptable risks, required 3rd party certification has been completed and User Acceptance Testing has been completed and the user group representative has signed off.

Final Communication plan

Your communication plan should clearly set out what is going to be communicated to whom at what interval or trigger point. Define a list of finite items for tracking only add to the list critical issues that arise.

Switch from weekly to daily reporting. You may have done this already, but if this is included in your release plan the critical players will know what to expect. This can increase confidence in the upcoming release.

Make sure to target the right people and the right info.

All test team members should be issue aware. Because there are a limited number of critical issues being tracked, this should not be overwhelming to individual team members. It should be possible for any test team member to report on the current status when needed and help them to focus on the important issues for release.

In addition to this all management levels should be statistic aware with details available. The best reporting is high level with drill down information available. Particularly in situations where there will be outstanding issues, or test cases left in a “failed” state, or not run at all, a drill down provides the space for explanations that can address management concerns.

Post implementation test walk through – there’s no second chance.

It is critical that the post implementation testing be thoroughly planned. A walk through of the plan should be on the schedule.

In your release walk through make certain that you spell out and cross check all of the details of the post implementation testing. Everything that you need must be in place when you are releasing new code into a live environment. Be sure to cover:

Materials

Environments

Test cases – review your proposed post release test cases, make sure they’ve already been tested in the test environment.

safe test data – make sure there’s no chance of accidentally using actual real world data. Dummy data in the test system is not necessarily safe in the live environment.

Back out plan – participating in the project level plan

Find out if the baseline code is available to check any concerns

Are you ready?