How to Add Robots.txt and Sitemap.Xml

Robots.txt
Each search engine have its own crawlers like robots, scooters etc. When these crawlers goes to any site first of all they check the robots.txt file and then crawl the whole site. According to robots.txt file permission they crawl a website. For example if i restrict the google crawler to contact us page then my contact us page is not crawled. This is an idea about Robots and why we use the Robots.txt file. Here is the following code that is used for robots.txt file.

user-agent:*
Disallow:



Use * and leave blank field in front of disallow if you want to allow all robots to crawl your site. You can restrict a page by using following code.

user-agent: *
disallow: /cgi/

This code give the instruction to crawlers not to crawl a page with cgi name.

How to upload it in website

After completing the coding of Robots.txt in Notepad, Save it with robots.txt name. Now Use FTP server to upload it and upload it in the root directory of your website.

What is Sitemap.xml
Sitemap.xml and sitemap.html are two types of sitemaps. Sitemap.Xml is a type of sitemap that is used only for crawlers. Sitemap is a file that make a site easy to visit. When crawlers come to website then according to sitemaps they crawl the whole site. Sitemap.xml can be created by using various online tools like http://www.xml-sitemaps.com/. Enter your website url here and download the coding file. Save the same file in notepad wiht sitemap.xml file and upload it in Root directory of your website.