Curl

This algorithm pulls a webpage via URL and returns as text

(no tags)

Algorithmia Platform License · Internet Access

Try the API

"User-agent: *\nDisallow: /search\nAllow: /search/about\nDisallow: /sdch\nDisallow: /groups\nDisallow: /index.html?\nDisallow: /?\nAllow: /?hl=\nDisallow: /?hl=*&\nAllow: /?hl=*&gws_rd=ssl$\nDisallow: /?hl=*&*&gws_rd=ssl\nAllow: /?gws_rd=ssl$\nAllow: /?pt1=true$\nDisallow: /imgres\nDisallow: /u/\nDisallow: /preferences\nDisallow: /setprefs\nDisallow: /default\nDisallow: /m?\nDisallow: /m/\nAllow:    /m/finance\nDisallow: /wml?\nDisallow: /wml/?\nDisallow: /wml/search?\nDisallow: /xhtml?\nDisallow: /xhtml/?\nDisallow: /xhtml/search?\nDisallow: /xml?\nDisallow: /imode?\nDisallow: /imode/?\nDisallow: /imode/search?\nDisallow: /jsky?\nDisallow: /jsky/?\nDisallow: /jsky/search?\nDisallow: /pda?\nDisallow: /pda/?\nDisallow: /pda/search?\nDisallow: /sprint_xhtml\nDisallow: /sprint_wml\nDisallow: /pqa\nDisallow: /palm\nDisallow: /gwt/\nDisallow: /purchases\nDisallow: /local?\nDisallow: /local_url\nDisallow: /shihui?\nDisallow: /shihui/\nDisallow: /products?\nDisallow: /product_\nDisallow: /products_\nDisallow: /products;\nDisallow: /print\nDisallow: /books/\nDisallow: /bkshp?*q=*\nDisallow: /books?*q=*\nDisallow: /books?*output=*\nDisallow: /books?*pg=*\nDisallow: /books?*jtp=*\nDisallow: /books?*jscmd=*\nDisallow: /books?*buy=*\nDisallow: /books?*zoom=*\nAllow: /books?*q=related:*\nAllow: /books?*q=editions:*\nAllow: /books?*q=subject:*\nAllow: /books/about\nAllow: /booksrightsholders\nAllow: /books?*zoom=1*\nAllow: /books?*zoom=5*\nAllow: /books/content?*zoom=1*\nAllow: /books/content?*zoom=5*\nDisallow: /ebooks/\nDisallow: /ebooks?*q=*\nDisallow: /ebooks?*output=*\nDisallow: /ebooks?*pg=*\nDisallow: /ebooks?*jscmd=*\nDisallow: /ebooks?*buy=*\nDisallow: /ebooks?*zoom=*\nAllow: /ebooks?*q=related:*\nAllow: /ebooks?*q=editions:*\nAllow: /ebooks?*q=subject:*\nAllow: /ebooks?*zoom=1*\nAllow: /ebooks?*zoom=5*\nDisallow: /patents?\nDisallow: /patents/download/\nDisallow: /patents/pdf/\nDisallow: /patents/related/\nDisallow: /scholar\nDisallow: /citations?\nAllow: /citations?user=\nDisallow: /citations?*cstart=\nAllow: /citations?view_op=new_profile\nAllow: /citations?view_op=top_venues\nAllow: /scholar_share\nDisallow: /s?\nAllow: /maps?*output=classic*\nAllow: /maps?*file=\nAllow: /maps/api/js?\nAllow: /maps/d/\nDisallow: /maps?\nDisallow: /mapstt?\nDisallow: /mapslt?\nDisallow: /maps/stk/\nDisallow: /maps/br?\nDisallow: /mapabcpoi?\nDisallow: /maphp?\nDisallow: /mapprint?\nDisallow: /maps/api/js/\nDisallow: /maps/api/staticmap?\nDisallow: /maps/api/streetview\nDisallow: /mld?\nDisallow: /staticmap?\nDisallow: /maps/preview\nDisallow: /maps/place\nDisallow: /help/maps/streetview/partners/welcome/\nDisallow: /help/maps/indoormaps/partners/\nDisallow: /lochp?\nDisallow: /center\nDisallow: /ie?\nDisallow: /blogsearch/\nDisallow: /blogsearch_feeds\nDisallow: /advanced_blog_search\nDisallow: /uds/\nDisallow: /chart?\nDisallow: /transit?\nDisallow: /extern_js/\nDisallow: /xjs/\nDisallow: /calendar/feeds/\nDisallow: /calendar/ical/\nDisallow: /cl2/feeds/\nDisallow: /cl2/ical/\nDisallow: /coop/directory\nDisallow: /coop/manage\nDisallow: /trends?\nDisallow: /trends/music?\nDisallow: /trends/hottrends?\nDisallow: /trends/viz?\nDisallow: /trends/embed.js?\nDisallow: /trends/fetchComponent?\nDisallow: /trends/beta\nDisallow: /musica\nDisallow: /musicad\nDisallow: /musicas\nDisallow: /musicl\nDisallow: /musics\nDisallow: /musicsearch\nDisallow: /musicsp\nDisallow: /musiclp\nDisallow: /urchin_test/\nDisallow: /movies?\nDisallow: /wapsearch?\nAllow: /safebrowsing/diagnostic\nAllow: /safebrowsing/report_badware/\nAllow: /safebrowsing/report_error/\nAllow: /safebrowsing/report_phish/\nDisallow: /reviews/search?\nDisallow: /orkut/albums\nDisallow: /cbk\nAllow: /cbk?output=tile&cb_client=maps_sv\nDisallow: /maps/api/js/AuthenticationService.Authenticate\nDisallow: /maps/api/js/QuotaService.RecordEvent\nDisallow: /recharge/dashboard/car\nDisallow: /recharge/dashboard/static/\nDisallow: /profiles/me\nAllow: /profiles\nDisallow: /s2/profiles/me\nAllow: /s2/profiles\nAllow: /s2/oz\nAllow: /s2/photos\nAllow: /s2/search/social\nAllow: /s2/static\nDisallow: /s2\nDisallow: /transconsole/portal/\nDisallow: /gcc/\nDisallow: /aclk\nDisallow: /cse?\nDisallow: /cse/home\nDisallow: /cse/panel\nDisallow: /cse/manage\nDisallow: /tbproxy/\nDisallow: /imesync/\nDisallow: /shenghuo/search?\nDisallow: /support/forum/search?\nDisallow: /reviews/polls/\nDisallow: /hosted/images/\nDisallow: /ppob/?\nDisallow: /ppob?\nDisallow: /accounts/ClientLogin\nDisallow: /accounts/ClientAuth\nDisallow: /accounts/o8\nAllow: /accounts/o8/id\nDisallow: /topicsearch?q=\nDisallow: /xfx7/\nDisallow: /squared/api\nDisallow: /squared/search\nDisallow: /squared/table\nDisallow: /qnasearch?\nDisallow: /app/updates\nDisallow: /sidewiki/entry/\nDisallow: /quality_form?\nDisallow: /labs/popgadget/search\nDisallow: /buzz/post\nDisallow: /compressiontest/\nDisallow: /analytics/feeds/\nDisallow: /analytics/portal/\nDisallow: /analytics/uploads/\nAllow: /alerts/manage\nAllow: /alerts/remove\nDisallow: /alerts/\nAllow: /alerts/$\nDisallow: /ads/search?\nDisallow: /ads/plan/action_plan?\nDisallow: /ads/plan/api/\nDisallow: /ads/hotels/partners\nDisallow: /phone/compare/?\nDisallow: /travel/clk\nDisallow: /hotelfinder/rpc\nDisallow: /hotels/rpc\nDisallow: /flights/rpc\nDisallow: /commercesearch/services/\nDisallow: /evaluation/\nDisallow: /chrome/browser/mobile/tour\nDisallow: /compare/*/apply*\nDisallow: /forms/perks/\nDisallow: /shopping/suppliers/search\nDisallow: /ct/\nDisallow: /edu/cs4hs/\nDisallow: /trustedstores/s/\nDisallow: /trustedstores/tm2\nDisallow: /trustedstores/verify\nDisallow: /adwords/proposal\nDisallow: /shopping/product/\nDisallow: /shopping/seller\nDisallow: /shopping/reviewer\nDisallow: /about/careers/apply/\nDisallow: /about/careers/applications/\nDisallow: /landing/signout.html\nDisallow: /webmasters/sitemaps/ping?\nDisallow: /ping?\nDisallow: /gallery/\nDisallow: /landing/now/ontap/\nAllow: /searchhistory/\nAllow: /maps/reserve\nAllow: /maps/reserve/partners\nDisallow: /maps/reserve/api/\nDisallow: /maps/reserve/search\nDisallow: /maps/reserve/bookings\nDisallow: /maps/reserve/settings\nDisallow: /maps/reserve/manage\nDisallow: /maps/reserve/payment\nDisallow: /maps/reserve/receipt\nDisallow: /maps/reserve/sellersignup\nDisallow: /maps/reserve/payments\nDisallow: /maps/reserve/feedback\nDisallow: /maps/reserve/terms\nDisallow: /maps/reserve/m/\nDisallow: /maps/reserve/b/\nDisallow: /about/views/\nDisallow: /intl/*/about/views/\n\n# Certain social media sites are whitelisted to allow crawlers to access page markup when links to google.com/imgres* are shared. To learn more, please contact images-robots-whitelist@google.com.\nUser-agent: Twitterbot\nAllow: /imgres\n\nUser-agent: facebookexternalhit\nAllow: /imgres\n\nSitemap: http://www.gstatic.com/culturalinstitute/sitemaps/www_google_com_culturalinstitute/sitemap-index.xml\nSitemap: http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml\nSitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml\nSitemap: https://www.google.com/sitemap.xml\n"

Install & Use

Use

curl -X POST -d '"http://www.google.com/robots.txt"' -H 'Content-Type: application/json' -H 'Authorization: Simple YOUR_API_KEY' https://api.algorithmia.com/v1/algo/util/Curl/0.1.1
  
Curl Docs →