I've developed a Twisted Python crawler that does something very similar to that. The possibility that it would work well seemed dubious at first, but I've been pleasantly surprised with the results.

Email me at [email protected] - I'd be happy to discuss the technology and progress I've made so far.