Separate names with a comma.
Discussion in 'Search Engine Optimization' started by webguru, Feb 17, 2018.
What's the significance of Robots.txt file in a website?
Roborts.txt file tells the search engines which pages to access and index on your website and which pages not to.
Robots.txt file is a file which can help search engines to find the sitemap of your site. It also lets them to know which pages to ignore while crawling a website (not to index). The robots.txt file is a file which is kept at the root folder to tell search engines where they can and cannot go on your page indexing.