This morning I completed a small project to build an RSS feed crawler in Scheme. My main purpose was to exercise a recent integration we've done of Sleepycat's DbXml database to Scheme. The integration isn't ready for general distribution yet--it's still pretty rough around the edges--but I'd gladly make it available to others who are interested in experimenting with it.

The program reads configuration from on DbXml collection (think of it as a table) and then grabs the RSS feeds for each feed in the configuration. All of the channel information is inserted in one collection and the items are inserted in another, after being tidied up.

We did the integration using the FFI (foreign function interface) collection and a shim to translate DbXml's C++ API into C functions. We're not handling exceptions yet and some of the functions in the API aren't completely supported. I'm also not confident that the pointers are being registered properly for garbage collection. And, of course, there's documentation to write. Probably the biggest to-do, however, is to think carefully through what the right idioms are for using a native XML database in Scheme. Right now, its just the C++ API, for the most part, and that's not the most natural thing to use in a functional programming language.