Systems & Software Talk 

No Customer Will Ever Do That!

Probable MPAA Rating of PG-13 because a street synonym for excrement is used.

I have to believe that this blog-title provides common ground for many of us in this business. I think it binds us all together in uncontrollable ROTFL.

Who are the people uttering such a statement? Are these people smarter than we are? Is Lex Luthor secretly populating the planet with a breed of marketing people who have been taught to contribute one and only one thing to a design review meeting?

People at QA Forums inquire often about Tester-to-Developer ratio. I often wonder about the ratio of Title-Utterance  TO  Customer-Actually-Did-Do-That.  My own experience suggests it is 1 : >10. What does your own data tell you?

ABOUT THIS ENTRY

This explores a missed opportunity at design time to prevent a production performance issue. It also illustrates the potential cost savings against the background of actual costs in terms of immediate losses.

CREDITS

Much credit has to go to those people who fought long and hard for cultural change, some of which resulted in getting me into a design review meeting to begin with, where simian representatives were not normally included. "Help me Dr. Zira!"

SETTING

Come join me in a past design review of a motherboard for a factory-hardened Programmable Logic Controller (PLC). The seven characters were indeed characters. Their eyes extruded resentment at me, "What is a lowly tester doing here in the midst of some of the greatest brains this company has to offer?" "Shouldn’t you be off eating bananas?" "If there isn’t duct tape across your labrum, there should be."

CONTEXT MOTHER

… board that is. This design review was intended to approve a motherboard pilot product. Of major concern was the Zilog Z80 allocated to handle all five communications channels. Two channels were intended to support Ethernet. One channel was intended to drive a proprietary high-speed communications channel. One channel was for RS-232 and the last channel was intended to support RS-488. Someone indicated that the shortage of real estate on the motherboard was the primary reason for selection of the smaller 4-MHz Z80, over the large space-gobbling 25-MHz 68020. It was at that moment that I closed my eyes to see lots of little flashes of blue light resulting from increased neural transmitter activity. Or was that glow the glow of a waning Vodka presence from the night before? Anyway, I opened my eyes and the duct tape loosened its grip on my lips. I stammered a bit and said, "I don’t think a Z80 will handle all these channels at full-demand." I walked to the white-board with some of my scratchy hand calculations, thinking, "Hey, I could be a doctor writing prescriptions!" All the eyes in the room tried to laser pain to my legs in order to prevent me from arriving at the white-board. I made it! I transferred my papered prescription for performance disaster to the white-board and walked those skeptical sets of eyes through the calculations. Two techie designer heads bobbed while trying to remain inconspicuous to the person with the power – Mr. Magisterial Marketer!  Referring to driving all comm. channels full-bore and to no one's surprise, Mr. Magisterial Marketer exclaimed, "No customer will ever do that!"

LACUNA TO CALCULATE

• Average cost burden (not to be confused with pay) to company = $80/hour
• Duration of meeting = 1.5 hours
• Attendees = 7
• Cost to change board = $3,500.00, estimated.
• Not counting the cost of 68020 vs. Z80 = 150% of Z80. This change would have cost roughly $4,400.00. One of course would need to calculate long term costs of the 68020 and reconcile that against projected sales; and blah blah blah. It might be that one would discover that to be a moot point after reading this.
• Other costs and data:
  • Duct tape = $1.50
  • Number of people wishing I was off eating bananas = 6.
  • Cost of me eating a pound of bananas in those days = $0.17.

MONTHS LATER

A customer from down under was feeling down under apparently. This customer called one of our executives and gave some background information and then said (as the story goes), "Get this shit out of here and get me something that works!" Do you need to know what this customer was talking about? Probably not! Mr. Magisterial Marketer obviously forgot to include release notes that instructed the customer that despite having 5-comm channels, one could not rely on them to perform when most needed.

I will abandon the story at this point and move on to costs.

PAUSE TO CALCULATE

• Palliate-the-Customer Entourage Travel Expenses = > $100,000.00
• Repair/Replacement/workaround costs = ~$20,000.00

COST SUMMARY (Concept courtesy of Master Card)

• Cost to repair the performance issue at design time = $4,400.00
• Cost of post-production performance issue (not counting negative word-of-mouth long-term impacts) = > $120,000.00
• The look on Mr. Magisterial Marketer’s face when I would later encounter him and say to him, "Sixty eight Oh Twenty!" = Priceless!

LESSONS LEARNED

• Despite having factual evidence, improper decisions can still prevail.
• The almighty dollar prevails over common sense and factual data.
• The statement ""No Customer Will Ever Do That!" carries clout when spoken by a marketing person.
• One can do "Performance Testing" early on. In this case, "Performance Testing" was executed on paper.

JAKE YOU TALK ABOUT CALCULATIONS AND DON’T TELL US HOW

What’s up with that?

I could spend the next four pages summarizing. I will keep it well under that with an overview, and within context. Some of this of course is telling you what time it is with your own watch! Bare-the-facts with me!

• A CPU uses clock cycles to fetch and execute instructions.
• Not all instructions carry equal weight in terms of clock cycles required.
• The instructions I identified used from one to four clock cycles per.

In this case, I had done some prep-work in advance. I acquired some open source Z80 assembler code that was used to empty an inbound communications buffer. I walked one byte through the code in order to identify both the count and the instructions used. From this I added up the clock cycles. I made an assumption that the outbound transmit logic would be roughly the same. My numbers were thus doubled. I calculated these out for each communication channel and arrived at a figure in a specific unit of time (required clock-cycles per second) that suggested at least four Z80s at 20-MHz or a screaming 68020 at 25-MHz. The party was not over at this point as I had to determine the communication interrupts impact on the main engines. I no longer remember or care to remember those details. Anyway, I concluded that the main engines were capable of handling the potential interrupt rate.

RELEVANT DATA

Not all assemblers are equal in terms of how many instructions are actually assembled at "assemble" time.

Bus speeds and on-board caching may have an impact on similar calculations or create latency. To what extent if any, I am no longer in tune with this low-level world – by choice.

WHY WAS THE CUSTOMER MAD?

The PLC could not keep up with the network demand interrupts to process like items flying by on a conveyor belt. OUCH!

DID YOU EVER TEST THIS PRIOR TO RELEASE?

Did I at some point actually test this prior to it going out the door? Yes. It behaved as I expected - poorly. I filed the appropriate high-severity defects.

FOR A VISUAL DEMONSTRATION OF INTERRUPTS

http://www.sqablogs.com/JakeBrake/178/Easy+Interrupt-Handling+Demo.html

Untitled Comment

11:52, 2006-Mar-25  ..  Posted by WorkingHorse     

1:~2 for where I work

I think people make the difference between what ratio will do the job, 1:1 or just 1:3 ratio. For each one (Devs) of them tripping (flat faced on the cement ground) on their own arrogance and ignorance is a day of laughter and rejoice for all the quality minded QA around the globe. If it affects the company and your food stream get them fired (it only takes one bad product to ruin the reputation).