The Analysis Engine is a distributed, massively scalable RESTful web crawler married to a huge bank of proxy servers and a super-extensible data processing pipeline.

I helped design it and build it.