Web Crawler : Review of Different Types of Web Crawler, Its Issues, Applications and Research Opportunities

Main Article Content

Keyur Desai
Virala Devulapalli
Prof. Smita Agrawal
Prof. Preeti Kathiria
Dr. Atul Patel

Abstract

Today's search engines are equipped with dedicated agents known as “web crawlers†keen to crawling large web contents online which are analyzed and indexed and make the content available to users. Crawlers act together with thousands of web servers over periods expanding from weeks to several years. These crawlers visits several thousands of pages every second, includes a high-performance fault manager, are platform independent or dependent and are able to get used to a wide range of configurations without including additional hardware. This paper is focused on prerequisites of crawler, process of crawling and different types of crawlers. This paper give review about some potential issues related to crawler, applications and research area of web crawler.

Keywords: search engine, web crawler, www, Indexing, website analysis,

Downloads

Download data is not yet available.

Article Details

Section
Articles