Web scraping (also known as web harvesting, web mining or web data extraction) is the act of using programming to extract information from the web.

Web scraping works by requesting HTML pages from a website and extracting specific data by taking advantage of patterns in the HTML markup, or by embedding a fully-fledged web browser. More advanced systems of web scraping, namely with regards to magnitude, scheduling, and automation, are often referred to as spiders, or web crawlers.

Potential uses include:

• Retrieving product or stock prices comparison for comparison,

• Contact scraping and collecting email addresses,

• Site mashup or building an alternative front-end for an existing site,

• Collection of real-estate pricing or auto sales statistics,

• Website change detection

• Building archives of dead pages

The practice of Web scraping has drawn a lot of controversies because the terms of use or copyrights for some websites and electronic publications do not allow certain kinds of data mining. While web scraping is not illegal on its own, legal issues can arise if being done with malicious or plagiaristic intentions, to circumvent a site's purchasing system or subscription fees, or other fraudulent or maligned purposes.

There have been numerous cases of lawsuits and other legal actions against companies and individuals. Before attempting to extract any information from a website in a method that is potentially contrary to the sites' indented usage, it is important to exercise due diligence in educating yourself in applicable local and international laws as well as the site's terms of service, copyrights, and trademarks. Further discussion on legal implications is can be found online including Wikipedia, Hacker News and Laws.com.

Web crawling is a component of web scraping across multiple sites, indexing information on the web using a bot or "spider" and is a universal technique adopted by most search engines while honoring exclusion requests such as those published a robots.txt file place on the site.

In contrast, web scraping focuses more on the transformation of unstructured data on the web, typically from HTML into a structured form that can be more easily stored, manipulated, and analyzed using tools such as a database or spreadsheet.

Screen scraping has a similar purpose but involves the programmatic collection of visual data from a source (as opposed to parsing data as in Web scraping) and originally involved reading the terminals memory or video data by connecting the terminals to another computer's input port.

web-scraping is most often tagged along with:

   ➡ ^{python       ( including beautifulsoup, scrapy and selenium )}
   ➡   ^{javascript      ( including node.js and phantomjs )}
   ➡ ^{r              ( including rvest )}
   ➡ ^selenium
   ➡ ^{xml          ( including xpath )}
   ➡ ^{java          ( including jsoup )}
   ➡ ^php
   ➡ ^{vba          (including vba-excel)}

A note on spelling

The verb is spelled to scrape, or as the present participle scraping, and is not to be confused with to scrap or scrapping, which is to discard something you no longer want or need, or to not continue with a plan.

Further Reading:

• Wikipedia: Web scraping
  ^{Overview of the types of web scraping, as well as techniques, software, legalities and prevention.}

• GitHub: Guide to Preventing Web Scraping
  ^{Detailed advice on prevention of web scraping. (Original article on Stack Overflow)}

• HartleyBrody: I Don't Need No Stinking API
  ^{A commercial blogger's views, advice and tips for scraping}

• Stack Overflow: Scraping data from website using VBA
  ^{Discussion and examples of getting started scraping with VBA}

• SitePoint: Web Scraping for Beginners
  ^{Theory and examples for beginner to web scraping}