Test Ideas Workshop, Example output
As a comment on my Test Ideas entry, Joe Strazzer asked if there were any examples of the output. Here is a cut down and sanitised version of output from a workshop for a change to existing functionality.
Change Requirement: The current system forces users to print out documents on their machine once a transaction is complete. Failure in the printer means that the user must call up the Help Desk to complete the activity. The change was to allow the users to complete the task by hand, on pre-printed form, without the need for calling the Help Desk. This required additional information to be shown on the screen to allow the user to complete the pre-printed form.
Reason for the Test Ideas workshop. On the face of it, this is a simple “does it show the correct information” test, however, there are various states that the system can be in which causes some complexity, hence the workshop.
Workshop output:
|
ID |
Source |
Contact |
Description |
Size |
Importance |
|
1 |
Usage Scenarios |
Tester A |
Exchange manual for system generated docs |
M |
H |
|
2 |
Usage Scenarios |
Tester B |
If Abort is selected, is there any 'Go Back' facility? |
S |
M |
|
3 |
Usage Scenarios |
Tester C |
Can user return to main menu from Printer Failure screens? |
S |
M |
|
4 |
Quality Factors |
Tester D |
Filling in docs manually does not affect |
M |
M |
|
5 |
Quality Factors |
Tester A |
Screen layout can accommodate all necessary info e.g. Lists |
M |
H |
|
6 |
Quality Factors |
Tester A |
Display/ordering of info on screen should be such that it helps user to write documents |
S |
L |
|
7 |
Failure Modes |
Tester B |
If printer fails and system doesn't register this, can user resort to the Help Desk screen facility, I.e. is the option to choose the Help Desk screen available at meaningful points during the process? |
M |
H |
|
8 |
Failure Modes |
Tester B |
What happens when the user dismisses the Printer Failure screens accidentally? Can they go back? |
S |
M |
|
9 |
Failure Modes |
Tester B |
What happens when the connection to host (via modem) is lost before option to print/view Printer Failure screen is presented/selected |
S |
H |
|
10 |
Failure Modes |
Tester A |
What happens if the user elects to abort DURING printing |
S |
H |
|
11 |
Failure Modes |
Tester A |
What happens if a reprint fails - can the user return to the Printer Failure screen? |
S |
M |
|
12 |
Failure Modes |
Tester C |
What happens if the screensaver or some other timeout kicks in whilst user is on Printer Failure screen - can the user return to the Printer Failure screen? |
S |
H |
|
13 |
Failure Modes |
Tester C |
Master/Slave combos. Is the user directed to the correct screen? |
M |
H |
|
14 |
Failure Modes |
Tester D |
Mended printer now works |
S |
M |
|
15 |
Requirements |
Tester A |
All user functions to be tested. |
L |
M |
|
16 |
Requirements |
Tester A |
All types of docs to be tested |
L |
H |
|
17 |
Requirements |
Tester A |
Sub-types of docs to be tested |
S |
M |
|
18 |
Requirements |
Tester B |
Printer Failure screens must provide all details required by the user in order to continue processing without printing |
L |
H |
|
19 |
Requirements |
Tester B |
Printer Failure screens should not be accessible if print not aborted |
S |
H |
|
20 |
Requirements |
Tester B |
Works for all types of output |
L |
H |
|
21 |
Requirements |
Tester C |
Different types of user devices |
M |
H |
|
22 |
Requirements |
Tester C |
Welsh docs |
L |
H |
|
23 |
Requirements |
Tester D |
Any MIS info recorded? |
M |
L |
|
24 |
Requirements |
Tester D |
All types of results checked |
L |
H |
Some of these tests are obvious and any tester on their own would have picked them out. The ones that may be less obvious and may not have been picked out without the workshop are IDs 2, 4, 8, 9, 10, 13, 19.
Items of interest are:
2) This one is a question that was raised in the workshop as it was not covered in the requirements. This is not unusual for a question to be raised and the question can then be referred back to the BA for resolution.
12) Screensavers are particular problem with our application as, by design, they kick the user out of the process they were executing. This circumstance is not discussed in the requirements, so is a good test to try.
19) Not explicit in the requirements, but knowledge of the system told us that this would cause serious problems.
22) Some of our users are in
23) Again, not stated in requirements, so does the user want any MIS info? We needed to pose the question. Would we have asked this question if we hadn’t had the workshop? Maybe, maybe not.
It is worth noting that these ideas are high level and are not in a state that you can execute tests from, but that is not the point of the workshop, the point is to get ideas that can then be expanded upon. Some of the ideas above you can see what the test would consist of, some require some major thinking about the details of the test.
You might say that ID 5 should have been in the “Requirements” category rather than “Quality Factors” and you could be right, but it doesn’t matter, what matters is getting the ideas and if they are mis-categorised, then so what? The categories are only there to prompt you for ideas, they are not important for the testing.
As a result of the testing, 6 bugs were found in 10 days of testing, 3 of them serious. Would we have found them without the workshop? There is no way of telling, but I feel very comfortable about the level of testing we performed on this change as a result of the workshop. 6 bugs for 10 days does not seem like a very good return, but there was a lot of setup for the tests and we were working on a stable base for the change, so I was not too disappointed.
