One of the best decisions I made when we started Kynetx was to use test-driven development from the start. Our core engine, written in Perl, has thousands of tests and we run them religiously as we develop and before we deploy a new version to the servers. When someone reports a bug, the first thing we do is write a test to exercise it. That way we know we really have a bug and we know when it's squashed. As a result, we've done over a thousand deploys of the rules engine over the last 6 years and have only truly broken production a few times.

Interestingly, most of the time when production has broken, we kinda of expected it to. Not that we wanted it to, but we knew the changes we so vast and far reaching that it was almost certain that something would happen we hadn't anticipated and written a test for. For those kind of changes, you can't be so afraid of breaking production that you stifle innovation.

With all that, I'm ashamed to say that we've never really developed a test-driven development process for KRL, the rules language that is at the heart of everything we do. Partly, that's because for 4.5 of the six years that we've been writing KRL, the rulesets have been simple—usually less than 5 or 6 rules—and rarely interacted with each other. Over the last year or so, however, we've been writing applications in KRL that are much more complex than what we did previously. Consequently, knowing that they work correctly after a change can be a challenge...unless you have tests.

Over the past few weeks I've been trying to rectify this. We're doing a large project in KRL, arguably the most complicated pico structures and event-query APIs that we've built to date. I knew that the only way to make it rock solid was have tests and that meant developing the tools and processes necessary to effectively write them.

The first attempt at this is documented on the KRL wiki: Test-Driven Development and KRL. We have a KRL Test module to aid in writing and running tests along with a test harness in Perl.

I've used this to write tests for a few important modules and we'll be working on more of CloudOS over the next while. Ultimately we hope to have as good of test coverage on CloudOS as we do on the underlying engine so we can develop in confidence and reduce the friction of chasing bugs that should have never made it into production.

The tools we have now are simple. No doubt they'll gain additional features over time. If you're a KRL developer, feel free to offer suggestions. Better yet, send a pull request.