What's The Significance Of Robots.txt File In A Website?

What's the significance of Robots.txt file in a website?

 

Roborts.txt file tells the search engines which pages to access and index on your website and which pages not to.

 

Robots.txt file is a file which can help search engines to find the sitemap of your site. It also lets them to know which pages to ignore while crawling a website (not to index). The robots.txt file is a file which is kept at the root folder to tell search engines where they can and cannot go on your page indexing.