"Testing isn't an event" says John Michelsen, the Chief Architect and a co-founder of iTKO, Inc. I was the moderator on an InfoWorld Webcast this afternoon where John presented. SOA brings new challenges to testing. Testing individual services is similar to code-level testing in any other development effort, but testing integration points, especially when there are dozens of hundreds of them is more difficult.
One issue is that business analysts and QA folks need to be involved in the testing process. To make that more difficult, SOA testing is something that needs to happen continuously.
If the business side, QA, production operators, and support people aren't involved in your iterative development process, you're not really doing agile development, you're still doing waterfall. Even though you're iterating in development, you're not involving the rest of the organization. On the other hand, involving these other groups in the iterative process can lower the cost per test as well as giving faster discovery and resolution of problems.
John spent some time going over iTKO's LISA test product and showing how it supported complete, collaborative, and continuous testing. LISA allows you to define actions and then assert the desired results. This gives a declarative testing model so you don't write code. LISA uses WSDL to show the valid operators in a Web service; you build tests against that list.
You can take the same workflows that you build for a functional test and then stage them in a load mode and find performance bottlenecks. You can build baselines and then compare changes as services are maintained in various stages of the lifecycle.
SOA is marked by constant change at the component and app level. This requires that we move beyond the kinds of traditional operational systems (disk, server load, etc.) Web services management system provide some of this, but it's really monitoring and recovery. I can see how a comprehensive and continuous testing system could happily live along side the traditional operations platforms and the newer Web services management systems to augment those with continuous functional and performance coverage.
