web

web / AnalyzeURL / 0.2.17

README.md

Table of Contents

  1. Introduction
  2. Examples
  3. Credits

Introduction

Given a url and a target string, returns several pieces of information on the url, including "text" - the text of the page (similar to /util/Url2Text) , "title" - the page title, "summary" - a summary of the page text (using either the meta description of the page, or, if this is not available, the summary of the page text via /nlp/Summarizer), "thumbnail" - the url of the main image on the page, if it can be located, and a boolean denoting if the page source contains the target string. Note that all information is provided on a best effort basis, if something can be found it will not be included in the output.

Input:

  • (Required): The URL of the page to analyze
  • (Optional): An optional search term to look for in the page

Output:

  • Summary of the web page.
  • Date of the page if available.
  • Thumbnail of the page if available.
  • Has an optional search term been used?
  • Text content of the web page.
  • Title of the web page.
  • URL of the web page.
  • Received status code for requesting the given URL.

Examples

Example 1.

  • Parameter 1: Sample investopedia article..
[
   http://www.investopedia.com/articles/professionaleducation/10/6-steps-becoming-a-real-estate-agent.asp
]

Output:

{  
   "summary":"Thinking of becoming a real estate agent? Take these steps seriously and you'll have your first \"sold\" sign up with your savings account still intact. ",
   "date":"2010-08-03T11:00:00-06:00",
   "thumbnail":"http://i.investopedia.com/inv/genericcontentimages/personalfinance/200275051-001.jpg",
   "marker":true,
   "text":"Becoming a successful real estate agent is a combination of investing time in education, researching a broker who can help you get your first clients and passing state and national licensing exams. But that's not all there is to the industry. Read on to find out some of the more overlooked aspects of the real estate business. No matter in which state you live, you must take pre-licensing courses. However, state requirements differ greatly. For instance, California requires three college-level courses. Others (such as Idaho, which requires two courses totaling 90 hours) require a set number of hours of education. Contact your state's real estate commission for your state's requirements for licensing. Some real estate agencies have specific education requirements. Thus, you may have to take an additional course after being hired on with an agency. A real estate brokerage is the agency or office from which real estate agents and brokers work. Since working with a broker is a requirement in order to practice as a real estate agent, you will need to contact a broker before graduating from your training course. Brokers have at least three years additional real estate training, and can guide you through questions you have when it comes to working in the field, as well as listing and selling homes. When you look for a broker, think about size of brokerage, its reputation and additional training offered. Check broker reputations by reading online comments, asking friends and neighbors who they've had experiences with and getting advice from your instructor on choosing a brokerage. Another way to learn more about a brokerage is by carefully crafting your interview questions. This will not only help you gather information, but solid interview questions help the broker determine if you'd fit in well with the agency. A few questions to ask: Which answers are acceptable is up to you. For instance, one person may prefer a brokerage that does ask for additional coursework because of a desire for more training before jumping in, while another may appreciate having someone who will work with them every day while learning. Real estate licenses require the passing of state and national exams. In addition, you may have to provide a criminal background check. Between the courses, exam and license fees for a real estate salesperson, you can expect to pay at least $200, though prices vary from state-to-state. While becoming a real estate agent isn't cheap, it's cheaper than entering many professions. Startup fees are estimated between $1,500-2,000, which should be divided between licensing courses, business cards, signs and advertising and association fees – not counting additional exam fees. Since real estate is a commission-based business, you'll also need enough money set aside for you to get by for a few months. These are approximations of actual costs because they can vary based on individual choices and state-by-state costs. In order to utilize the title \"realtor,\" you must join the National Association of Realtors (NAR). This is done by choosing an affiliated brokerage as well as attending a set number of meetings designated by your local chapter. The best way to build your portfolio is twofold: get a mentor, and use your personal network. Barbara Kennon, of the National Association of Realtors, says the best arrangement for a new agent is to find a mentor in the real estate agency you choose who guides you towards buyer/seller contacts and splits commission. You'll learn the profession from your mentor, while gaining your first commission checks. Also, asking your friends and family for referrals of people who are considering buying or selling a home is a great way to begin networking. Someone's always looking for a new home, and that referral may get you started in your new business. Becoming a real estate agent is similar to starting a small business. Even though you'll work within a brokerage of established realtors or real estate agents, you need a startup fund for business expenses and to cover several months of personal expenses while you build your client base. Take every step seriously, and you'll have your first \"sold\" sign up with your savings account still intact. Want to learn how to invest  Delivered twice a week, straight to your inbox.",
   "title":"6 Steps to Becoming a Real Estate Agent  | Investopedia",
   "url":"http://www.investopedia.com/articles/professionaleducation/10/6-steps-becoming-a-real-estate-agent.asp",
   "statusCode":200
}

Example 2.

  • Parameter 1: Sample Techcrunch article page.
  • Parameter 2: Search term (algorithm) to look for in the page.
[
  "http://techcrunch.com/2015/03/12/algorithmia-launches-with-more-than-800-algorithms-on-its-marketplace/",
  "algorithm"
]

Output:

{
  "summary": "Algorithmia, the startup that raised $2.4 million last August to connect academics building powerful algorithms and the app developers who could put them to..",
  "date": "2015-03-12 07:55:03",
  "thumbnail": "https://tctechcrunch2011.files.wordpress.com/2015/03/screen-shot-2015-03-11-at-8-05-08-pm.png?w=764&h=400&crop=1",
  "marker": true,
  "text": "Algorithmia, the startup that raised $2.4 million last August to connect academics building powerful algorithms and the app developers ... Latest headlines delivered to you daily",
  "title": "Algorithmia Launches With More Than 800 Algorithms On Its Marketplace  |  TechCrunch",
  "url": "http://techcrunch.com/2015/03/12/algorithmia-launches-with-more-than-800-algorithms-on-its-marketplace/",
  "statusCode": 200
}

Credits

This algorithm is built on top of the following algorithms: util/Url2Text & nlp/Summarizer.