Regression testing is more interesting when I know I'm going to find problems.
We're working a on a new release, and there are several sources of destabilization: a new version of the operating system potentially affects timing-sensitive stuff, a new platform with a faster processor definitely puts timing-sensitive stuff at risk, and we've integrated a big chunk of new functionality that was off on its own branch for a long time, with the expected merge difficulties. The basic happy-path functionalities are mostly working just fine, and the exceptions have pretty straightforward problems with straightforward fixes.
Around the edges, well, there are some problems. In particular, I've had a couple of hits on a very-intermittent D channel outage that I thought was fixed, many months ago. This is worrying. Is this the same problem, and it wasn't really all the way fixed before? Or something else that just happens to produce the same symptom? So far I've been testing mostly on the new platformm - is the problem on the new platform only, or will it also hit the older hardware? Also, I've uncovered one of those annoying "the channels don't all cleanup properly" problems when I tear down a conference with a lot of connections and a particular combination of features in use, which my developer says is going to be a major pain to fix. I'm sure I haven't found all the weirdnesses yet, and so far I've barely touched that new mode of behavior that got added, because we need to make sure the standard product will be shippable first. |