(untitled) (International Journal of Advanced Research in Computer Science, 2017)
Today’s search engines are equipped with dedicated agents known as â€œweb crawlersâ€ keen to crawling large web contents online which are analyzed and indexed and make the content available to users. Crawlers act together with thousands of web servers over periods expanding from weeks to several years. These crawlers visits several thousands of pages every second, includes a high-performance fault manager, are platform independent or dependent and are able to get used to a wide range of configurations without including additional hardware. This paper is focused on prerequisites of crawler, process of crawling and different types of crawlers. This paper give review about some potential issues related to crawler, applications and research area of web crawler. Keywords: search engine, web crawler, www, Indexing, website analysis,