What is the state of the art technology in crawling?
Mozenda is an interesting SaaS that allows you to build “agents” for scraping and crawling.
See also my quick and dirty webpages:
· 100 Best Web Crawler Videos – Meta-Guide.com
· Best Mozenda Videos – Meta-Guide.com