How to use robots.txt file?

Discussion in 'Search Engine Optimization' started by Chanthini, May 23, 2013.

  1. Chanthini

    Chanthini
    Expand Collapse
    Junior Member

    Joined:
    Jan 31, 2012
    Messages:
    22
    Likes Received:
    1
    User-agent: *
    Allow: /
    Disallow: /Some-folder/
    Disallow: /some-folder/folder1

    And
    User-agent: *
    Disallow: /Some-folder/
    Disallow: /some-folder/folder1

    which one is correct way? Please explain how it will be work.
     
  2. taylor swift

    taylor swift
    Expand Collapse
    Junior Member

    Joined:
    Jul 30, 2012
    Messages:
    113
    Likes Received:
    1
    it's simple go to html or editor part of you website and past
    <meta name="robots" content="index, follow" />
    1)robot means google spider which index you all website post to google search engine.
    2)index means allow google to index
    3)follow means when ever you post a new it index it
     
  3. jammesscot

    jammesscot
    Expand Collapse
    Junior Member

    Joined:
    Apr 12, 2013
    Messages:
    51
    Likes Received:
    1
    Second one is little bit better than first one and there is no need to mention the Allow: / if you won't use disallow command to any folder or any link still it will automatically be allowed. you can use this as follows and sub folder will also be disallowed by the following instructions .

    User-agent: *
    Disallow: /Some-folder/
     
  4. theindemnitybroker

    theindemnitybroker
    Expand Collapse
    Junior Member

    Joined:
    Jun 12, 2013
    Messages:
    9
    Likes Received:
    0
    Robots txt file can be used websites security purposes.which type of file can not be indexed google
     
  5. Chanthini

    Chanthini
    Expand Collapse
    Junior Member

    Joined:
    Jan 31, 2012
    Messages:
    22
    Likes Received:
    1
    Hi Guys, Thanks for your Great information.
     
  6. vijay raj

    vijay raj
    Expand Collapse
    Junior Member

    Joined:
    Jun 4, 2013
    Messages:
    35
    Likes Received:
    3
    Robots.txt is important in SEO to crawl all web pages faster and quickly index all search engines
     
  7. vijay raj

    vijay raj
    Expand Collapse
    Junior Member

    Joined:
    Jun 4, 2013
    Messages:
    35
    Likes Received:
    3
    All search engines are identifying websites only robots.txt file. It helps to analyze the complete website frequently
     
  8. myoled2

    myoled2
    Expand Collapse
    Junior Member

    Joined:
    Jun 19, 2013
    Messages:
    5
    Likes Received:
    0
  9. propestkc

    propestkc
    Expand Collapse
    Junior Member

    Joined:
    Jun 19, 2013
    Messages:
    2
    Likes Received:
    0
    Robot.txt file is located in the main server. First of all you have to access the main root link of the server to use the robot.txt file. It is little bit hard to access the robot.txt file.

    www.propestkc.com
     
  10. buypinehurst

    buypinehurst
    Expand Collapse
    Junior Member

    Joined:
    Jun 19, 2013
    Messages:
    5
    Likes Received:
    1
    robots txt file helps security purposes and also what are pages google cahed or not cahed pages that can be filtered

    Pinehurst homes
     
  11. pestcontrolsarasota

    pestcontrolsarasota
    Expand Collapse
    Junior Member

    Joined:
    Jun 19, 2013
    Messages:
    4
    Likes Received:
    0
    search engine find the website link using the robot.txt file
     
  12. treeservicesnovimi

    treeservicesnovimi
    Expand Collapse
    Junior Member

    Joined:
    Jun 20, 2013
    Messages:
    4
    Likes Received:
    0
    robots
    .
    txt file
    to give instructions about their site to web robots. To use this you have to access the main server
     
  13. semaphore.v

    semaphore.v
    Expand Collapse
    Junior Member

    Joined:
    Jul 29, 2013
    Messages:
    54
    Likes Received:
    0
    Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. It must be in the main directory because otherwise user agents (search engines) will not be able to find it.
     
  14. Alex_Thompson

    Alex_Thompson
    Expand Collapse
    Junior Member

    Joined:
    Sep 13, 2013
    Messages:
    59
    Likes Received:
    0
    It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.
     
  15. samfrank

    samfrank
    Expand Collapse
    Junior Member

    Joined:
    Sep 18, 2013
    Messages:
    21
    Likes Received:
    0
    The robots.txt file is a very essential file if you wanna increase your site ranking.

    Robots.txt file

    When a search engine crawler comes to your website, it will look for a special root file on your website. That file is called robots.txt and it tells the search engine spider, which Web pages of your website should be indexed and which Web pages should be ignored.
     
  16. pryidevsblog

    pryidevsblog
    Expand Collapse
    Junior Member

    Joined:
    Aug 8, 2013
    Messages:
    110
    Likes Received:
    8
    Second one is right. you can use this. to give the privilege to crawlers.
     
  17. Manoj Nagesh

    Manoj Nagesh
    Expand Collapse
    Junior Member

    Joined:
    Dec 21, 2011
    Messages:
    52
    Likes Received:
    8
    This one is correct way...

    User-agent: *
    Allow: /
    Disallow: /Some-folder/
    Disallow: /some-folder/folder1
     
  18. anupam sharma

    anupam sharma
    Expand Collapse
    Junior Member

    Joined:
    Sep 7, 2013
    Messages:
    105
    Likes Received:
    4
    Second technique for adding robots is good and more clear..
     

Share This Page