Robots.txt is a text file we put on our site to tell search robots which pages you would like them not to visit.

____________________
SEO in swindon

robots.txt basically tells search engine spiders not to index certain areas of your site. For example, f you have web versions and print versions of your pages, you don't want both to show up in searches, so put the forprint/ directory in robots.txt Also you can protect private areas of sites from being visible to search engines, and thus to everyone.

The robots.txt file is used to instruct search engine robots about what pages on your website should be crawled and consequently indexed.

Robots text file is the file where you can allow or disallow user agent means robots or spiders to crawl or not to crawl that web page which you restricted. It is not necessary but good to have.

_____________________
Cloud Computing Energy Efficiency