What are the benefits for putting robot file in server ?

Please explain what is the benefit of putting robot file in detail . ?

 

The Robots.txt file is a small text document that you place in the Root Directory of your website server (where your website files are stored).

This file is one of the first things that a Search Engine "robot" crawler will look for when it comes along to index/crawl your website.

All of the legitimate Search Engine "crawlers" Spiders will obey the Robots.txt file; but now and then, you will find that a "robot" has crawled through one of the pages that you have "disallowed".

This is because some Search Engines do not look for the Robots.txt file; or they have been programmed to ignore them. Sphider (OpenSource) Search Engine is a good example of a search engine that will ignore the Robots.txt file if the person who is hosting the script has "told" programmed it to do so.

 

Robots.txt file use on server initially checked by the search engine crawlers (robots), and based on it search engine decides, which pages have to do index, and which pages do not have to index.