I’ve got an ongoing partially unresolved inquiry into converting sitemaps into mindmaps , which in ideal conditions would involve crawling a website, parsing the directory structure into a nested outline, and then inserting the crawled webpage titles into the outline. A cloud based (web) API would be the ideal form for this. I can further recommend my Meta Guide webpage, “100 Best Web Crawler Videos” .
What are some really interesting web crawling projects?
(Visited 407 times, 1 visits today)