Visual Studio 2010 test tools: Lessons from the trenches
Our QA team has been using the Visual Studio 2010 test tools (including Microsoft Test Manager) for several months now, and we have gone through a couple of releases with the tools. While the tools generally work well, it feels at times like we are beta testing for Microsoft. I have logged several bugs against the Test Manager, to Microsoft's credit they have been responsive to those. We also have found situations where the tools do not work the way expect them, and have had to adapt our processes to match or work around Microsoft's thinking about how the tools should be used. Here are a few items that might be of interest to others who are using or considering using these tools:
- Test Manager does not allow carraige returns in a test step. Here is a way to work around that. Write your test cases in Excel and use the Test Case Migrator Plus tool to import your test cases. Migrator Plus imports the test cases as they were formatted in Excel, including carraige returns in the test steps, if any. If you edit the test steps later in Test Manager, you cannot add additional carriage returns, however.
- Microsoft assumes that test case work items and bug work items reside in the same TFS project, if you want to be to able to submit bugs from the Test Manager, you have to set up your projects in TFS such that your test cases and bugs are in the same project. Some features will work across TFS projects, but many will not.
- Similarly, if you want to use the Recommended Tests feature where Test Manager will recommend which tests to re-run, your source code and your bugs will have to be in the same project.
- When linking Issue (or other bug type) work items to Test Case work items, DO NOT use the Test Case link type. This is for linking Test Cases to Shared Steps and will not work correctly when linking Issues to Test cases. We use the Tested By link type on the bug work item side, this creates a Tests (verb) link type on the Test Case side of the link. As in "this test case TESTS this bug, this bug is TESTED BY this test case".
- If you associate your test plans in Test Manager with TFS builds, be aware that the default retention policy for TFS builds is to retain only the last 10 builds, and also to delete the test results for any deleted builds. If you want to retain your old test results indefinitely, make sure you set the build retention policies accordingly. Otherwise your builds and test results will vanish out from under you, and you will scratch your head wondering where did my test results go?
- Only one suite of automated tests can be run from an instance of Test Manager at one time. In Quality Center it was possible to launch multiple sets of automated tests at the same time. Test Manager does not work that way. Something to keep in mind if you are coming from using Quality Center and QTP.
- Coded UI tests work well generally, but to make them flexible you have to use test parameters in your linked test cases in Test Manager. SearchProperties are also a must to learn to use effectively. These enable a test to be changed at runtime from what it was recorded, without changing the recording or UI Map. Microsoft assumes that you will re-record the tests when the application changes, this is a naive assumption, but the above features make that largely unnecessary.
- When setting up a TFS build server, you can run the build service as a Windows service or an interactive process. If you want to run your automated tests at build time, you will need to run the build as an interactive process. I do not run the automated tests are run time, so I run it as a service which builds the automated tests but does not run them at build time.
In general I would say that the Microsoft testing tools are not bad for a first generation set of tools. The MSDN forums and developer blogs will be very helpful in understanding what to expect and to ask for help. If you find a bug, the Microsoft Connect web site is where you want to log it.