1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

What's The Significance Of Robots.txt File In A Website?

Discussion in 'Search Engine Optimization' started by webguru, Feb 17, 2018.

  1. webguru

    webguru
    Expand Collapse
    Registered Member

    Joined:
    Feb 13, 2018
    Messages:
    24
    Likes Received:
    1
    What's the significance of Robots.txt file in a website?
     
  2. johnsimth

    johnsimth
    Expand Collapse
    Junior Member

    Top Poster Of Month

    Joined:
    Nov 2, 2017
    Messages:
    153
    Likes Received:
    1
    Roborts.txt file tells the search engines which pages to access and index on your website and which pages not to.
     
  3. Raj Sharma

    Raj Sharma
    Expand Collapse
    Junior Member

    Joined:
    Apr 10, 2017
    Messages:
    69
    Likes Received:
    2
    Robots.txt file is a file which can help search engines to find the sitemap of your site. It also lets them to know which pages to ignore while crawling a website (not to index). The robots.txt file is a file which is kept at the root folder to tell search engines where they can and cannot go on your page indexing.
     

Share This Page