Systems & Software Talk 

"Flipper"

A Performance Test Horror Story

Probable MPAA Rating of PG-17

[light_southernUSA_twang_on]

This ain’t about a dolphin. It is about mammals; mammals with supposed higher cognitive skills than other mammals on this here planet. The main mammal Gino filled every inch of his six-foot-six 300-pound frame with intimidation. Gino was the president of our 3^rd party software supplier. I’ll jis call them Slam Dunk fer purposes of this story. Gino was mad at me. At my five-ten 170 pounds, I took him seriously. He was mad acuz he didn’t like the fact that I didn’t like our contract with his company to supply us "robust" factory automation software. Hell, we wuz closer to Mars than his software was to robust. Scratchy-your-back/my-back influence was abundant in the contract our company made with his. Our company paid ½ cost of all software from Slam Dunk before we actually had an opportunity to test. (Slam Dunk was 'upposed ta have dun thayze own tesin too.) The other half was paid when we released the stuff. I guess that might be standard and nothin’ worth whining about. Fortunately our company had the presence of mind to actually have its own mammals test the stuff. I was one of those mammals.

Gino was even madder than his usual self about the latest release, because I was flipping and ranting about it; as expressed in pods of defect reports. This stuff was ungood. At best it could be called hackware. I had just run it through a simple smoke test named "Flipper". Flipper became famous and made it into all sorts of documentation throughout the company as well as the documentation that shipped with the product. Flipper was actually quite simple. People talked about Flipper! There were of course more tests, but Flipper had gained notoriety. It became the benchmark test. I was labeled as someone who practiced black magic. These people did not like me, ‘cuz I suppose I was always affecting their bottom line. Flipper integrated some key functionality to test integrated components. If you have ever seen a graphical display of a vat filling in a factory, for example; you can probably relate easily. Many of these systems afford the integrator with the opportunity to draw/redraw screen objects, bucket fill, or change colors based upon parameters being acquired from the host itself, or of other devices (PLCs) over a factory network – Ethernet (802.3 and/or 802.4 where needed). This can be done at both design and run time. Flipper consisted of doing a one-16-bit word write and read to a PLC via Ethernet. This should not have been a problem ever. The spec called for just one of these operations per second. Easy, except for this system! Flipper was just a simple read write, tied to a one-second timer. I used the simplest of graphics to represent the state on the monitor; a red square if bit zero of the designated word in the PLC equaled one and green if the opposite. If the timer expired and the bit in the word hadn’t flipped, I had the timer interrupt handler log an error. In the very first version of this hackware (1.0), the Flipper test showed that spec was met about 90% of the time. Pitiful, eh? But the execs had already decided to ship it despite my protests and objective evidence. Well, a few months after 1.0 went out the door, we were informed 2.0 would soon arrive. Gino’s side-kick Harvey decided that it was a travesty that the host computer was not running with Berserkely’s real-time extension to Eunuchs (Unix for purists). Slam Dunk called Harvey their Eunuchs guru. We, on my team called him an *******. Harvey must have gotten this idea real time from his dumb dog that he brought into his office everyday, acuz no higher-order mammal would ever do such a thing on short notice and without extensive research and experimentation. To me, this was kinda lahk taken the undercarriage of a VW beetle and deciding that it was a better fit under a 66 Cadillac Coupe De Ville. Anyway, Harvey was gittin’ rich off hackware. He took a vacation aboard his own personal yacht during the time the time this crisis was gathering cumulonimbus clouds about it.

So I dun got this new release. Load her up Jake! Launch Flipper! Damn, I couldn’t believe it! Flipper was showing severe degradation compared to version 1.0 to the tune of being capable of only one read/write every seven seconds nominally with best times at six seconds. Extensive research and analyses followed. I rewrote Flipper multiple times. Slam Dunk reps came in and tried to tune Eunuchs. Harvey was sailing happy and rich in the Atlantic somewhere. I was hoping that the real Flipper would use its bottlenose to punch a small hole in the hull of Harvey’s boat. Well, I dun filed the defects. Gino got madder and wanted to come see a live demo of this. So he and his entourage and some of our own execs scheduled a request for a demo of Flipper by me. You gotta understand our lab. It was a lab with all kinds of hardware – one-arm-clawed robots, them ovens used fer heat/hoomiditty testing, shock and vibration stuff, and ESD stuff. That’s probably why my hair kinked up into an Afro that would make Linc of Mod Squad envious. (As soon as pictures are up-load-able – I will upload a picture of that Fro). Well, the entourage showed up decked out in coat and tie. Gino had his other sidekick Frank, with him. Frank had brought a patch with him to fix the performance to bring it to sub-second – better than ever! Frank took a few minutes to install it and rebooted the system. It started up with arrogance, to Frank’s satisfaction.

Frank’s presence despite his pale and weakened slim look (turn sideways and he disappears), put fear into us. Charlie Manson would have been envious of that wild look on Frank's face. Frank said, "Let’s get started. Turn Flipper on." I looked over the entourage pod to see consensus and some helter-skelter. Gino asked me if I was sporting and wanted to wager against the patch that Manson installed. I stood up to Gino and asked, "Gino, if you are that confident, how about hanging your balls in the claw of that robot over there?" Pointing to the robot, I added, "I have it rigged so that if the performance degrades to beyond one second, the claw will close. If it stays at one-second or less, the claw stays open." Gino declined. The execs of my firm quizzed me with their oculus, wondering if I knew who all was present. Manson laughed and broke the ice. Everyone else laughed, probably out of fear of what Manson might do next.

The test failed. There was absolutely no improvement in performance. Six to seven seconds was the best the stuff would do. Gino looked at me with derogatory paint striped from ear to ear across his face. He said, "Black magic!" and walked away.

[light_southernUSA_twang_off]

The software shipped despite 43 showstoppers I had filed after getting through the entire release. Several execs huddled together and decided it was necessary to save Slam Dunk so they could work on the performance and get a new version to me shortly after Harvey’s return.

This software was soon blacklisted.

Learning for me:

1. Objective evidence is critical – at all times. Even though this shipped, I had evidence that could not be disputed. I did what I could and then it was out of my hands.
2. Objective evidence doesn’t always trigger proper thinking in higher-order mammals.
3. Could developers have devoted time to this upstream and discovered the same thing I did? Yes! Developers do not have performance testing exemption status. Much money in travel expenses could have been saved with a bit of diligence on their part.
4. Relative to testing, no matter how often you might get challenged, take the challenge – properly.
5. In any level or category of testing, always include a test that is a key indicator of performance.
6. Profit is the bottom line and sadly it took precedence over customer care.
7. A system must perform to spec and beyond. A customer will know immediately if it isn’t especially where factory automation is concerned.
8. Don’t be bullied by executives. Focus on showing them what they asked for.

Note! The robot was not connected to the PLC. Other than that - is this a true story? Yes.