Combine Cloud Algorithms to Build a Feature-Rich Website Mapper & Summarizer
Once upon a time, site mappers were arcane scripts which could take hours or days to examine a single website. But, thanks to scalable & interoperable cloud algorithms, it now takes only minutes… and includes a multitude of handy features powered by machine learning: auto-tagging, summarization, page-ranking, and more!
You may have already seen our Python tutorials for building a simple web-scraper or extracting structured data from a webpage, but today we’ll highlight a demo built entirely in JavaScript which combines five cloud-hosted microservices into a single useful tool:
- GetLinks recursively traverses a website of your choice, plotting them on a force-directed graph via D3
- PageRank examines the pages to create an ordered list akin to Google’s PageRank Algorithm
- Url2Text grabs the text from each page, allowing Summarizer to extract topic sentences while AutoTag generates keywords
All of this takes just a few simple lines of simple Javascript Code, and zero additional infrastructure. We invite you to try out the sitemapper demo and, if you like what you see, try building your own!
Map, summarize, and tag any website
Learn more in our Introduction to Microservices article.
Jon Peck
Developer Advocate for Algorithmia, helping programmers make the most efficient and productive use of their abilities.
Follow Me:
