Systems & Software Talk 

Performance "Testing" When Can it Start?

A long long time ago at a DoD Contractor far away…

I was initially assigned to a project as QA Manager. Initially my role was to fulfill the contractual obligations for both QA and Testing. This is when I learned about the differences. The USAF Program Office (PCO) told me I could not be both and then proceeded to educate me about why. That education about the differences is something I hold onto when opining in relevant QAF threads. The PCO asked me to choose the domain I wished to operate in and asked me to staff the vacant role I would leave behind. I conferred with the PM and found out that I would be responsible for staffing a team regardless of which role I chose. I chose Testing since the System Specification (SSS extra S for Segment) ) was loaded with all kinds of techie, geeky stuff that I couldn’t wait to tackle. I was strongly urged to handle the System & Systems Integration aspect, which by USAF definition included Performance Testing. "Bitchin!" ("Bitchin" had just superseded "boss" had just superseded "Heavy" had just superseded "Far Out (recycled))

I initially grew the test team to a total of four out of ultimately nine capable engineers and we set out to do our thing. We had documentation that would make most QAF members drool with envy. The governing document from which all else flowed pertinent to testing was the SSS. One key spec that caught my eye immediately was, "The system shall never exceed 65% CPU utilization during normal operations." Bitchin! (I will let you know when I mean bitching** as opposed to bitchin’) A performance-related spec! I was in binary heaven. And there were more specs of this category!

The design and development staffs consisted of technical leads only. They had to find 12 more designers/developers yet. We the Test Team were already planning! The contract had been awarded only two months prior.

The entire system (hardware and software) was to be built by this defense contractor. The system was intended to be the first of a new generation of Air-Intercept Control facilities. Ultimately, the system would be integrated with existing RADAR and telemetry systems. The system was to have the following as its processing "plant": Two DEC-VAX-11/780s, one with 2mB RAM, the other with 4mB RAM and a shared 4mB RAM solid-state unit. The main mission beside all the display bells and whistles was to track aircraft of course. In addition, the system was to provide on-demand intercept solutions for up to 35 controllers. Here is where a key performance spec came into play. This spec stated that, "The system shall provide real-time collision avoidance processing (CA) for up to 300 aircraft being tracked by the system." My immediate thought was "No Way!" Another key spec that this was to play along with was, "The system shall factor in winds aloft data (WAD) into all intercept solutions." By now, some of you might be thinking this is a lot of work for two 7.8 MHz systems. You are correct. I should note that all display processing was to be offloaded to dual-68020 display controllers interfaced to each control console. That would help.

I was due to present my System Test Plan at a formal Preliminary Design Review (PDR) two months after first seeing those specs. This plan according to PCO was to address how we would test this system in-house through all phases. The plan also had to address operational testing and evaluation (OPTEVAL). I felt I was in my element – for awhile. The CA spec gnawed at me. I conferred with several key people who advised me to get busy with making a case for PDR. I ran some desk calculations and kept coming up with data that showed there was no way this system could do the CA processing along with three RADAR and two high-speed telemetry interfaces.

PDR was here. In a boardroom full of 30 USAF personnel, their technical advisors, and about 20 of "us", the PDR progressed. My turn! I was a shy person at that age. I began talking while sitting. The PM next to me kicked my ankle and gestured me to stand. I was beet red and lacked any confidence in what I was about to present. As I talked, I waited for someone to stop me and ask me to leave. I was putting my entire career on the line. I was telling the USAF and their high-caliber technical advisors that the CA and WAD processing were too much and the system could handle WAD without CA for up to 100 aircraft, or – WAD and no CA. I presented the evidence and my confidence grew as I saw spec doubts dripping from several little splinter discussions. My blush was gone. One small group of advisors said they would take my analysis for their own and get back to us in a week. I had one minor faux pas a few minutes later when I presented the strategy for checking loss of secondary RADAR from aircraft during OPTEVAL. I said that we would have the aircraft bank steeply, wing down toward the RADAR in order to hide the aircraft transponder antenna. One USAF rep popped up and asked, "Why don’t you just have the pilot turn his transponder off?" I said, "Good idea! I like that better than my own." My face lit up RGB-red (255,0,0) in just under 0.65 seconds on that one. My anti-perspirant failed and the meeting adjourned.

As promised, the USAF delivered the response to my analysis. They approved a change to drop the CA spec. I felt relieved. I felt great! I had my job! Independent analyses agreed with my own! Bitchin'! ("Awesome" was still a decade away) This project however, was part of the project majority since it was a year late and had the usual cost-overruns! Keeping the CA would have added an order of magnitude to both those parameters. CA is found in most Air Traffic Systems today. Many question the reliability of it however. The processing logic is found both on the ground and aboard aircraft. It is still far from perfect and requires mucho compute horsepower. When it ultimately works in a robust fashion, it will be bitchin’ and the controllers now bitching** about CA today will exclaim, "Bitchin’!"

To me, this demonstrates:

1. System Testing or any testing, can start on day one,
2. "Testing" is not necessarily the act of "testing" per se, but involves examination and analyses, and ultimately – test and demonstration, and
3. The MIL-STD-483 "waterfall" was not really! (a friendly jab at the Agilistism-istic-ators! )
4. Bitchin' did not used to equal Bitching.

I was right

02:13, 2006-Mar-14  ..  Posted by philk10     

this blog is gonna be Bitchin !!

Good educational story very entertainingly written
More !!

super

07:20, 2006-Mar-14  ..  Posted by ovidiu     

Great past experience from which everybody can learn.