Bots are really software agents that zip across the web or other stores of data,sifting information and presenting them in an intelligent fashion. There are four characteristics which differentiate bots from normal programs ” intelligent,similarity to people,autonomy and communication.”The Web is unique both in scale and variety. It contains trillions of pages which are growing approximately by millions per day. And now you have bots in all areas from e-commerce to games,from tracking financial information to keeping virtual chat room active, from scouring the web for news to keeping track of auctions worldwide.Search Engines typically use programs that traverse the web and collect web pages so that they can be indexed,searched and presented to the user. Such programs are called ‘crawlers’ or ‘spiders’.
- Spider is a robotic browser like program that downloads webpages from all over the internet.
- Crawler is a wandering spider that automatically follows links found on different pages.
- Indexer is a blender like program that dissects webpages that are downloaded by spiders.
- The Database is a warehouse of the pages downloaded and processed.
- Search Engine Results : digs search results out of the database.
Most Popular Bots,Spiders and Crawlers List from Famous Websites with Descriptions:
Yahoo! Slurp – Yahoo Search crawler
Bingbot – Microsoft’s Bing webcrawler
Googlebot – Google Crawler
Baidubot – Chinese search engine
Askbot – Ask.com
AOLbot – AOL Search Engine
Related posts:
