How to use robots.txt file?

User-agent: *
Allow: /
Disallow: /Some-folder/
Disallow: /some-folder/folder1

And
User-agent: *
Disallow: /Some-folder/
Disallow: /some-folder/folder1

which one is correct way? Please explain how it will be work.

 

it's simple go to html or editor part of you website and past
<meta name="robots" content="index, follow" />
1)robot means google spider which index you all website post to google search engine.
2)index means allow google to index
3)follow means when ever you post a new it index it

 

Second one is little bit better than first one and there is no need to mention the Allow: / if you won't use disallow command to any folder or any link still it will automatically be allowed. you can use this as follows and sub folder will also be disallowed by the following instructions .

User-agent: *
Disallow: /Some-folder/

 

Robots txt file can be used websites security purposes.which type of file can not be indexed google

 

Hi Guys, Thanks for your Great information.

 

Robots.txt is important in SEO to crawl all web pages faster and quickly index all search engines

 

All search engines are identifying websites only robots.txt file. It helps to analyze the complete website frequently

 

robots file very useful for search engine what are file can be cached google that cab be blocked robots file

http://myoled.co.uk/gu10-led-lights

 

Robot.txt file is located in the main server. First of all you have to access the main root link of the server to use the robot.txt file. It is little bit hard to access the robot.txt file.

www.propestkc.com

 

robots txt file helps security purposes and also what are pages google cahed or not cahed pages that can be filtered

Pinehurst homes

 

search engine find the website link using the robot.txt file

 

robots

.​

txt file

to give instructions about their site to web robots. To use this you have to access the main server​

 

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. It must be in the main directory because otherwise user agents (search engines) will not be able to find it.

 

It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

 

The robots.txt file is a very essential file if you wanna increase your site ranking.

Robots.txt file

When a search engine crawler comes to your website, it will look for a special root file on your website. That file is called robots.txt and it tells the search engine spider, which Web pages of your website should be indexed and which Web pages should be ignored.

 

Second one is right. you can use this. to give the privilege to crawlers.

 

This one is correct way...

User-agent: *
Allow: /
Disallow: /Some-folder/
Disallow: /some-folder/folder1

 

Second technique for adding robots is good and more clear..