Web Crawler is a/an

A Web Crawler is a software application that is programmed to visit websites and read their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot".

Here are the steps a Web Crawler typically performs:

1. The crawler starts with a list of URLs to visit, called the seeds. The seeds can be a static list set up by the search engine company or dynamic URLs identified during past crawls.

2. The crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier.

3. URLs from the frontier are recursively visited according to a set of policies.

4. If the crawler is performing archiving of websites it copies and saves the information as it goes. The archives are usually stored in such a way they can be viewed, read and navigated as they were on the live web, but are preserved as ‘snapshots'.

5. The crawler will periodically return to the sites to check for any information that has changed. The frequency with which this happens is known as the crawl rate or crawl frequency.

6. The information that the crawler gathers is used to update a search engine’s index. This index is then used by the search engine to provide fast searches of the web.