A while ago I went to a security testing presentation, sponsored by the local (Greater Boston) ASQ chapter.  I've been to several such over the past few years (starting around 2002, I think), and they all muddle together in my mind as "you really ought to do security testing for your software, here are some cool and frightening demos of what will happen if you don't".  Well, the more recent ones include some suggestions for how you might go about it.  (Plus the implication of "if you want it done right, hire my company, or at least buy my book :)

This presentation was different.  The presenter talked about projects to gather data on security flaws - weaknesses in architecture, design, and implementation, and software attacks and attack patterns, and working on vulnerability theory, and protection schemes.

See http://makingsecuritymeasurable.mitre.org/ - there's a *lot* of stuff there.

Software security is growing up.  When I first became aware that there was such a thing as security testing, it was pretty clearly something I did not know how to do - it needed a special mindset, and my mind just doesn't twist that way.  But now, the people whose minds do twist that way seem to be starting to make sense of it, and write about it, in ways that will let people like me do useful security testing if/when I need to.  Cool.

And there are tools - here's a list from April 2007, that should still be mostly useful:
http://www.networksecurityjournal.com/features/open-source-security-tools-applications-resources-041007/

Looks like fun ...

We took to saying the process had vaporized, because it exited unexpectedly leaving no useful information behind.  It was repeatable, but not on demand - I would fire the load scenario and wait.  There was nothing useful in the log files, even if I turned the log level up.  There was no Dr Watson file (this is Windows 2003), and there should have been.  And, I couldn't make it die if I ran the process with a debugger attached.

We had an awful time with this one - my developers were pulling their hair out, and my developers are *good*.  They finally had to go to Microsoft support for advice, which eventually did the trick.

My secret identity is "Mostly-clueless-with-a-PC Girl", so I'd better make notes about this for future use.  (So what am I doing working in a Microsoft shop?  Well, the product is a conference bridge, and I do the telephony testing.)

(1) Make sure your entire process is subject to an exception handler, make sure there's an exception handler of last resort.

(2) The normal exception handlers are stack-based, therefore if your bug is trashing the stack, you may not get a Dr Watson file.

(3) Check your string handling and similar functions - might you be overwriting a buffer somewhere?  That's a prime candidate for stack overwrite.

(4) Might you be throwing an exception within an exception handler?  Repeatedly?

The advice from Microsoft support was to add a "vectored exception handler", which wrote the exception code and address to a file, then returned to commence the normal, stack-based exception handler. 

Quote from explanation that got passed around: "The idea is that the vectored exception handler gives us a chance to log some data before the complex SEH unwind begins and possibly goes haywire.  This API was added in XP/2K3, likely to provide a means to troubleshoot the scenario we are encountering.  Since SEH is stack based it is susceptible to severe malfunctions."

This exposed the information that trouble starts with an access violation from a bad read (from still-under-development proprietary hardware), then additional exceptions get thrown during the attempt to handle the first one.

Whew!  Useful information at last!  Developers added defensive code for the bad read(s), and are investigating why it happens in the first place.

Noticed in my local (Boston MA area) IEEE newsletter: a class offering, "Technical Writing for Technical Professionals".  This is not "how to be a tech writer", but is intended to help working engineers, scientists, and other techies improve their writing of technical documents.  Great idea, if it reaches even a few of the people who need it, the world will be a better place.

If this, or something similar, has been offered before, I haven't noticed it.  I've waded through a lot of really poorly written technical prose over the years, so I'm all for it.  A distressing number of my various developer colleagues seem to have believed that if they knew how to program, they didn't need to know how to write - and some of my tester colleagues seem to have believed that no one else was ever going to try to make sense of their test documents, so they didn't need to bother with writing stuff other people could read.  Some of the worst offenders were native speakers of English. 

A few years ago, I taught an in-house class for a previous employer that covered some of this type of thing - it was focused on reviewing specifications, but that's just the flip side of writing them.  No way of knowing if it helped much, unfortunately, as my class pretty much coincided with the beginning of the "let's move real development work away from this campus" thing for that employer, and the numbers of specs being produced dropped off a lot.

Once in a while, there's a bug that's really difficult to isolate.

Reproducible?  Oh yes - but not on demand.  The effect is really obvious (i.e., a process crashes), but it doesn't leave any useful traces behind - the core dump or Dr Watson file is consistently non-existent or corrupt, and the log files don't show anything significant.  Turning up the log level either changes the timing so the bug doesn't hit, or doesn't add any useful information. 

I haven't yet found a good way of dealing with these - the best I've been able to do is a slow and rather painful accumulation of peripheral information.  What's going on on the SUT when the bug hits?  What sort of problems can cause the process to exit without leaving tracks?  (Buffer overflows are a possibility ... maybe down in very low-level software.  And though BIOS or hardware problems are not the way to bet, they aren't out of the question.)

Some years ago, chasing one of these for a previous employer, the vital clue was provided by a coworker who noticed that the bug happened a specific time after a process failover - and that specific time delta matched a TCP timeout, which was an indication that some connection wasn't being cleaned up after the failover, which in turn gave the developer an idea of where to look.

So far, I've dealt with these only while a release was under development, never as a problem reported by a customer.  Good thing - I don't think there's any way to solve them quickly, and mistakes due to working under pressure can send you down false trails for a long time.