Breadth First Site Map

No algorithm description given

A more efficient site mapper that explores in a breadth-first fashion. Given a url and a max number of pages to return, it explores in a breadth-first fashion until it detects that is has passed the maximum allowed number of pages, or until it has run for two minutes, whichever comes first. Returns an adjacency map of urls. Optionally, you can provide a list of required terms as a third argument. If you do this, the algorithm will return a hashmap with two keys, "map", whose value is the site map as described above, and "marked", whose value is a list of URLs whose html (including human-readable text) contains at least one of the strings provided in the third argument. This is most useful for finding specific html tags, though be careful of very short or common words that might be present either in html or in human readable text.

Tags
(no tags)

Cost Breakdown

0 cr
royalty per call
1 cr
usage per second
avg duration
This algorithm has permission to call other algorithms which may incur separate royalty and usage costs.

Cost Calculator

API call duration (sec)
×
API calls
=
Estimated cost
per calls
for large volume discounts
For additional details on how pricing works, see Algorithmia pricing.

Internet access

This algorithm has Internet access. This is necessary for algorithms that rely on external services, however it also implies that this algorithm is able to send your input data outside of the Algorithmia platform.


Calls other algorithms

This algorithm has permission to call other algorithms. This allows an algorithm to compose sophisticated functionality using other algorithms as building blocks, however it also carries the potential of incurring additional royalty and usage costs from any algorithm that it calls.


To understand more about how algorithm permissions work, see the permissions documentation.

1. Type your input

2. See the result

Running algorithm...

3. Use this algorithm

curl -X POST -d '{{input | formatInput:"curl"}}' -H 'Content-Type: application/json' -H 'Authorization: Simple YOUR_API_KEY' https://api.algorithmia.com/v1/algo/web/BreadthFirstSiteMap/0.2.17
View cURL Docs
algo auth
# Enter API Key: YOUR_API_KEY
algo run algo://web/BreadthFirstSiteMap/0.2.17 -d '{{input | formatInput:"cli"}}'
View CLI Docs
import com.algorithmia.*;
import com.algorithmia.algo.*;

String input = "{{input | formatInput:"java"}}";
AlgorithmiaClient client = Algorithmia.client("YOUR_API_KEY");
Algorithm algo = client.algo("algo://web/BreadthFirstSiteMap/0.2.17");
AlgoResponse result = algo.pipeJson(input);
System.out.println(result.asJsonString());
View Java Docs
import com.algorithmia._
import com.algorithmia.algo._

val input = {{input | formatInput:"scala"}}
val client = Algorithmia.client("YOUR_API_KEY")
val algo = client.algo("algo://web/BreadthFirstSiteMap/0.2.17")
val result = algo.pipeJson(input)
System.out.println(result.asJsonString)
View Scala Docs
var input = {{input | formatInput:"javascript"}};
Algorithmia.client("YOUR_API_KEY")
           .algo("algo://web/BreadthFirstSiteMap/0.2.17")
           .pipe(input)
           .then(function(output) {
             console.log(output);
           });
View Javascript Docs
var input = {{input | formatInput:"javascript"}};
Algorithmia.client("YOUR_API_KEY")
           .algo("algo://web/BreadthFirstSiteMap/0.2.17")
           .pipe(input)
           .then(function(response) {
             console.log(response.get());
           });
View NodeJS Docs
import Algorithmia

input = {{input | formatInput:"python"}}
client = Algorithmia.client('YOUR_API_KEY')
algo = client.algo('web/BreadthFirstSiteMap/0.2.17')
print algo.pipe(input)
View Python Docs
library(algorithmia)

input <- {{input | formatInput:"r"}}
client <- getAlgorithmiaClient("YOUR_API_KEY")
algo <- client$algo("web/BreadthFirstSiteMap/0.2.17")
result <- algo$pipe(input)$result
print(result)
View R Docs
require 'algorithmia'

input = {{input | formatInput:"ruby"}}
client = Algorithmia.client('YOUR_API_KEY')
algo = client.algo('web/BreadthFirstSiteMap/0.2.17')
puts algo.pipe(input).result
View Ruby Docs
use algorithmia::*;

let input = {{input | formatInput:"rust"}};
let client = Algorithmia::client("YOUR_API_KEY");
let algo = client.algo("web/BreadthFirstSiteMap/0.2.17");
let response = algo.pipe(input);
View Rust Docs
import Algorithmia

let input = "{{input | formatInput:"swift"}}";
let client = Algorithmia.client(simpleKey: "YOUR_API_KEY")
let algo = client.algo(algoUri: "web/BreadthFirstSiteMap/0.2.17") { resp, error in
  print(resp)
}
View Swift Docs
Discussion
  • {{comment.username}}