2007-Apr-19 - Some metrics that make bad sense
Recently, the manager has shared with us some metrics that we will use in our performance evaluation this year. I list here the ones that I think may have side effects:
1. Not a defect / defects found < 5%
"Not a defect" is a category of QA's reported bugs. It not only includes duplicated cases and wrongly submitted cases, but also includes suggestions / feature requests.
This metric means to reduce duplicated or wrongly submitted cases.
Side effect: Discourage suggestions on improving the product.
2. 80+% bugs found before beta
This metric means to find as many bugs as early (before enter beta) as possible.
Side effect: Discourage bugs finding during beta phase.
3. P1 & P2 bugs in beta<3%
We have priority evaluation for each reported bugs. P1 means highest priority, which needs immediate fix, and P5 is the lowest.
This metric means to find as many CRITICAL bugs as early (before enter beta) as possible.
Side effect: Intendedly, conservatively underrate priority during beta phase. Thus developers may not pay enough attention to some real critical bugs found in beta.
4. 30% increase on test automation rate
There are several metrics like this one regarding automation test. These metrics mean to increase test execution efficiency.
Side effect: Discourage adding new test cases. Because once you add a new case, if you cannot make it automated, this rate will decrease. And many of the force error and exploration test cases are very hard or waste of time to turn into automation.
What metrics can we use then to reduce duplicated or wrongly submitted cases, encourage to find as many as critical bugs as early as possible, and increase test execution efficiency?
|
