Survey of Web Crawler Algorithms

Abstract

In today’s scenario, World Wide Web (WWW) is flooded with huge amount of information. Due to growing popularity of the internet, finding the meaningful information among billions of information resources on the WWW is a challenging task. The information retrieval (IR) provides documents to the end users which satisfy their need of information. Search engine is used to extract valuable information from the internet. Web crawler is the principal part of search engine; it is an automatic script or program which can browse the WWW in automatic manner. This process is known as web crawling. Crawling algorithms are crucial in selecting the pages that satisfies the users’ needs. This paper reviews the researches on web crawling algorithms used on searching.