Systems & Software Talk 

Manual Data-driven Test Procedure Example

Are both Blackbox and Whitebox testing necessary?

"Why do you go for White box testing, when Black box testing is available?" – from a post at SQAForums.

This is an excellent question with a vast domain of reasons why. I’ll venture some opinions and reasons.

ASSUMPTION:

Software development embodies more work than writing code. Software development includes debugging, unit, and unit-integration testing – at a minimum. Therefore, one can conclude that "white-box" testing is required and should not be a matter of choice or that one is optional as the original question might indicate.

IMO:

Omitted or neglected white box testing is simply deferred work; work remaining for others downstream in the life-cycle. Unfortunately in many organizations, where white box testing is neglected, it appears to management that development is further along than anticipated! When that happens, those forces that have the strongest influence on release dates will act to coerce a premature release based upon perceived progress – not real progress. In the interim, what typically will have happened? QC finds a large number of defects that should have been detected and corrected in development. This large number overwhelms the defect management process and people get crabby. R&D in some cases will tell the QC people to quit reporting so many defects. The product goes out the door. Waves of stupid questions arrive at the doorstep of QC. "Why did YOU let that bug get into production?" "Don’t YOU people test?" Then the QC people get really crabby and begin to commiserate at SQAForums.

     There are typically many areas of code untouchable by black box test engineers - if we say most of the testing executed by the black box test engineer is via the human interface. Here are a few examples of many of the possibilities:

1. Retry loops
2. Timing loops
3. Poorly written queries (debatable of course!)
4. Conversion of data with overlying formatting code

Here is a specific example from my past related to item 4. This example illustrates two defects, neither of which could be caught in black-box testing.

Context:

• ANSI-FORTRAN/77 (contract required ANSI-compliance)
• Back in the days when a numbered menu on a GUI was still vogue.
• The menu had to display 12 items in the current context of the application.

What?

• The developer used a library function to output the numeric list for the application context.
• The display guy took the output and displayed it.
• From a black-box point-of-view, "It" appeared to work great.

They did not unit test the code. I ended up doing the unit and unit integration testing for this piece. My original role was to develop software (before such software existed as it does today) to verify ANSI-compliance of project software source code. I found that the library function was non-ANSI and therefore in violation of the contract. This was thus an as-implemented bug undetectable via black-box testing. I concluded that there might be a fire under the smoke. I searched deeper. The code written by the display guy obscured a defect in the code (with the library call) that created the list. Apparently the display guy thought, "Oh well, that is how the non-ANSI function works." He took that output and formatted it properly for display. This layered defect was not detectable via black-box testing. Replacement of the non-ANSI library function call with custom code exposed the display-formatting defect.

LEARNINGS:

• Black-box testing cannot expose all defects.
• A robust test program requires testing in various categories and at many levels.
• Development means much more than writing code.

Functional Test Estimating Guide

Of course the referenced doc is heavily caveated!  :)  I prepared it as a reaction to a manager (go figure) who was confrontationally wondering why my team needed "so much time" to test (go figure again).  This soothed the manager. So, at the very least it can be applied as an eOintment to soothe a manager.

Please leave a comment to this topic if you download this document. By all means, please feel free to critique it, use it, rant and rave, shred and use it for confetti, or pour beer on it. If you find it useful and would like your comments incorporated for the benefit of others, please inform me. I will be happy to maintain a master document accessible here.

UPDATE: 15-JAN-2007

I find it amazing how many times this has been downloaded and neither critique nor gratitude has been communicated. The only way this can get better is with a feedback loop! So, let me ask again... If you download it, please leave a comment. If you have suggestions for improvement, please communicate them to me! Thank you!

http://www.sqablogs.com/uploads/j/JakeBrake/296.doc