What are some really interesting web crawling projects?
I’ve got an ongoing partially unresolved inquiry into converting sitemaps into mindmaps [1], which in ideal conditions would involve crawling a website, parsing the directory structure into a nested outline, and then inserting the crawled webpage titles into the outline. A cloud based (web) API would be the ideal form for this. I can further recommend my Meta Guide webpage, “100 Best Web Crawler Videos” [2].