Systems & Software Talk 

This is more about a human's performance and less about a digital computer's performance. The sailor in this story simply wanted to improve the TV Signal-to-Air Conditioner Noise ratio. He was watching American Graffitti and probably for the 50th time. Movie distribution to and circulation among the fleet was as efficient as Windows-XP. "Please wait - for a long time!" I pity modern day sailors stuck with a film such as "Crossroads", for months at a time.

When and Where: Aboard USS Midway CV-41 about 150 miles south of Tokyo Bay in 1981. Kim Carnes was singing about Bette Davis eyes. John Lennon was singing about something that I would soon be doing after the below ordeal – "Starting Over".

Why: Certification + Operational Test & Evaluation (OPTEVAL) of the Aircraft Carrier Landing System – a system designed to land aircraft automatically.

Who/Scope: Support came from many skill sets – RADAR technicians, Computer technicians, Systems Analysts, Pilots and the instrumented aircraft they ferried from Patuxent River, MD., aircraft technicians from the manufacturers, and on and on. They came from around the world. They came from Maryland, Virginia, California, Hawaii, The Philippines, and Japan. In other words, this was a multi-million-dollar undertaking.

System Background: Two computerized Precision Approach RADAR Air Traffic Controllers manned (PAR) consoles. Two vintage 1960’s era CP-848 (a.k.a. Univac 1219) computers allocated to do all the calculations, adjust for ship’s motion, wind-speed/direction, drive the PAR channels, the two displays, and - issue command/control information to the aircraft. For you MIL buffs – the Army-Navy nomenclature is/was AN/SPN-42 a.k.a. ACLS.

The computers were large. They had intake vents at the top front to suck in cold air from an air conditioning vent about a foot away. This was an interesting field modification to extend their operability beyond the inherent overheating problem that would occur at about 78 degrees F.

Univac 1219

The computers were in a compartment several compartments distant from the Carrier Air Traffic Control Center (CATCC) where the PAR was located. There was a Navy computer technician on duty here during operations.

My role: Coordinate all aspects of the operation to get the OPTEVAL accomplished.

Scenario: Aircraft would approach the ship from a holding pattern where they had been holding (burning precious fuel) until the ship was readied for operations. The weather at the time was less than ideal with rain and visibility of about 4 miles. The GO signal was issued. The aircraft (AC) commenced their approach and soon the first AC was acquired by the SPN-42 PAR at about 4 miles. The second AC was 2 miles behind and would soon be acquired by the other SPN-42 PAR channel.

"Flipper"

A Performance Test Horror Story

Probable MPAA Rating of PG-17

[light_southernUSA_twang_on]

This ain’t about a dolphin. It is about mammals; mammals with supposed higher cognitive skills than other mammals on this here planet. The main mammal Gino filled every inch of his six-foot-six 300-pound frame with intimidation. Gino was the president of our 3^rd party software supplier. I’ll jis call them Slam Dunk fer purposes of this story. Gino was mad at me. At my five-ten 170 pounds, I took him seriously. He was mad acuz he didn’t like the fact that I didn’t like our contract with his company to supply us "robust" factory automation software. Hell, we wuz closer to Mars than his software was to robust. Scratchy-your-back/my-back influence was abundant in the contract our company made with his. Our company paid ½ cost of all software from Slam Dunk before we actually had an opportunity to test. (Slam Dunk was 'upposed ta have dun thayze own tesin too.) The other half was paid when we released the stuff. I guess that might be standard and nothin’ worth whining about. Fortunately our company had the presence of mind to actually have its own mammals test the stuff. I was one of those mammals.

Gino was even madder than his usual self about the latest release, because I was flipping and ranting about it; as expressed in pods of defect reports. This stuff was ungood. At best it could be called hackware. I had just run it through a simple smoke test named "Flipper". Flipper became famous and made it into all sorts of documentation throughout the company as well as the documentation that shipped with the product. Flipper was actually quite simple. People talked about Flipper! There were of course more tests, but Flipper had gained notoriety. It became the benchmark test. I was labeled as someone who practiced black magic. These people did not like me, ‘cuz I suppose I was always affecting their bottom line. Flipper integrated some key functionality to test integrated components. If you have ever seen a graphical display of a vat filling in a factory, for example; you can probably relate easily. Many of these systems afford the integrator with the opportunity to draw/redraw screen objects, bucket fill, or change colors based upon parameters being acquired from the host itself, or of other devices (PLCs) over a factory network – Ethernet (802.3 and/or 802.4 where needed). This can be done at both design and run time. Flipper consisted of doing a one-16-bit word write and read to a PLC via Ethernet. This should not have been a problem ever. The spec called for just one of these operations per second. Easy, except for this system! Flipper was just a simple read write, tied to a one-second timer. I used the simplest of graphics to represent the state on the monitor; a red square if bit zero of the designated word in the PLC equaled one and green if the opposite. If the timer expired and the bit in the word hadn’t flipped, I had the timer interrupt handler log an error. In the very first version of this hackware (1.0), the Flipper test showed that spec was met about 90% of the time. Pitiful, eh? But the execs had already decided to ship it despite my protests and objective evidence. Well, a few months after 1.0 went out the door, we were informed 2.0 would soon arrive. Gino’s side-kick Harvey decided that it was a travesty that the host computer was not running with Berserkely’s real-time extension to Eunuchs (Unix for purists). Slam Dunk called Harvey their Eunuchs guru. We, on my team called him an *******. Harvey must have gotten this idea real time from his dumb dog that he brought into his office everyday, acuz no higher-order mammal would ever do such a thing on short notice and without extensive research and experimentation. To me, this was kinda lahk taken the undercarriage of a VW beetle and deciding that it was a better fit under a 66 Cadillac Coupe De Ville. Anyway, Harvey was gittin’ rich off hackware. He took a vacation aboard his own personal yacht during the time the time this crisis was gathering cumulonimbus clouds about it.

So I dun got this new release. Load her up Jake! Launch Flipper! Damn, I couldn’t believe it! Flipper was showing severe degradation compared to version 1.0 to the tune of being capable of only one read/write every seven seconds nominally with best times at six seconds. Extensive research and analyses followed. I rewrote Flipper multiple times. Slam Dunk reps came in and tried to tune Eunuchs. Harvey was sailing happy and rich in the Atlantic somewhere. I was hoping that the real Flipper would use its bottlenose to punch a small hole in the hull of Harvey’s boat. Well, I dun filed the defects. Gino got madder and wanted to come see a live demo of this. So he and his entourage and some of our own execs scheduled a request for a demo of Flipper by me. You gotta understand our lab. It was a lab with all kinds of hardware – one-arm-clawed robots, them ovens used fer heat/hoomiditty testing, shock and vibration stuff, and ESD stuff. That’s probably why my hair kinked up into an Afro that would make Linc of Mod Squad envious. (As soon as pictures are up-load-able – I will upload a picture of that Fro). Well, the entourage showed up decked out in coat and tie. Gino had his other sidekick Frank, with him. Frank had brought a patch with him to fix the performance to bring it to sub-second – better than ever! Frank took a few minutes to install it and rebooted the system. It started up with arrogance, to Frank’s satisfaction.

Frank’s presence despite his pale and weakened slim look (turn sideways and he disappears), put fear into us. Charlie Manson would have been envious of that wild look on Frank's face. Frank said, "Let’s get started. Turn Flipper on." I looked over the entourage pod to see consensus and some helter-skelter. Gino asked me if I was sporting and wanted to wager against the patch that Manson installed. I stood up to Gino and asked, "Gino, if you are that confident, how about hanging your balls in the claw of that robot over there?" Pointing to the robot, I added, "I have it rigged so that if the performance degrades to beyond one second, the claw will close. If it stays at one-second or less, the claw stays open." Gino declined. The execs of my firm quizzed me with their oculus, wondering if I knew who all was present. Manson laughed and broke the ice. Everyone else laughed, probably out of fear of what Manson might do next.

The test failed. There was absolutely no improvement in performance. Six to seven seconds was the best the stuff would do. Gino looked at me with derogatory paint striped from ear to ear across his face. He said, "Black magic!" and walked away.

[light_southernUSA_twang_off]

The software shipped despite 43 showstoppers I had filed after getting through the entire release. Several execs huddled together and decided it was necessary to save Slam Dunk so they could work on the performance and get a new version to me shortly after Harvey’s return.

This software was soon blacklisted.

Learning for me:

1. Objective evidence is critical – at all times. Even though this shipped, I had evidence that could not be disputed. I did what I could and then it was out of my hands.
2. Objective evidence doesn’t always trigger proper thinking in higher-order mammals.
3. Could developers have devoted time to this upstream and discovered the same thing I did? Yes! Developers do not have performance testing exemption status. Much money in travel expenses could have been saved with a bit of diligence on their part.
4. Relative to testing, no matter how often you might get challenged, take the challenge – properly.
5. In any level or category of testing, always include a test that is a key indicator of performance.
6. Profit is the bottom line and sadly it took precedence over customer care.
7. A system must perform to spec and beyond. A customer will know immediately if it isn’t especially where factory automation is concerned.
8. Don’t be bullied by executives. Focus on showing them what they asked for.

Note! The robot was not connected to the PLC. Other than that - is this a true story? Yes.