It is 2.30 am in the morning right now. I am in a good mood. The reason is that I just finished coding and testing the basic D-HarvestMan prototype for a single master, single slave configuration. And it works! The master was able to successfully bootstrap the slave crawler with a new domain and let it start downloading files from it. Hip hip hooray!
D-HarvestMan is a project to write a distributed crawler on top of the existing HarvestMan. Distributed programming is always exciting, and a distributed crawler is even more so :-)