Is robots.txt file necessary

waytovietnam · 03-26-2009, 02:04 AM

Hi
Is it necessary to put robots.txt file in the root directory of website, i know it is helpful to control crawlers, but if i not put this file in the website, does this cause some problem, please advice
Thanks for your support

**RedMatrix** · 03-27-2009, 02:02 PM

There would be no problem to your site if you did not have a robots.txt file.

**rockinaway** · 03-28-2009, 07:47 PM

Yup, no problem at all. Although I would recommend you have one personally to control the spiders, but yeah, shouldn't be a problem.

**Moelman** · 03-29-2009, 06:05 PM

I don't see why you wouldn't want to use one though, certainly only helps.

**rockinaway** · 03-29-2009, 07:56 PM

Strangely enough, I read something recently that said you shouldn't have a robots.txt file because the more areas you exclude from spiders, the less frequently they will visit. Is there truth to that?

**Moelman** · 03-29-2009, 08:00 PM

Hmm, I have not heard that myself, you don't happen to have a link to where you heard that do you? My word is certainly not always 100% but I think it's safe to use one. Can help with duplicate content issues as well, for example a printer-friendly version of a page or archives.

**rockinaway** · 03-29-2009, 08:19 PM

Damn, I can't find the actual link, but I am pretty sure I read it somewhere for SEO optimisation. It did shock me a bit, because I always thought it was better to use one.

**Trexle** · 04-03-2009, 05:47 PM

it guides the SE's what to index and what not to .. simple!!!!

**Niche** · 04-03-2009, 08:10 PM

I don't use robots file on most of my sites and I still get high position in the SERPs

**Brandon Sheley** · 04-04-2009, 05:32 PM

You don't "have" to have one, but it's strongly suggested
unless every page on your site, you want crawled
if it's just an html site, this might be okay, but any scripts or whatnot, I would add one.

03-26-2009, 02:04 AM	#1
waytovietnam Lurker Join Date: Nov 2008 Posts: 10	Is robots.txt file necessary Hi Is it necessary to put robots.txt file in the root directory of website, i know it is helpful to control crawlers, but if i not put this file in the website, does this cause some problem, please advice Thanks for your support

03-27-2009, 02:02 PM	#2
RedMatrix daviddamian.com Join Date: Feb 2006 Location: back in TX Posts: 4,043	There would be no problem to your site if you did not have a robots.txt file. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Free 3 day course

03-29-2009, 06:05 PM	#4
Moelman The Webmaster Join Date: Feb 2006 Location: Michigan, USA Posts: 3,328	I don't see why you wouldn't want to use one though, certainly only helps. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - The Largest LucasArts Gaming Community on the Internet To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - tell me your comments! To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Your #1 place to discuss all things World of Warcraft.

03-29-2009, 08:00 PM	#6
Moelman The Webmaster Join Date: Feb 2006 Location: Michigan, USA Posts: 3,328	Hmm, I have not heard that myself, you don't happen to have a link to where you heard that do you? My word is certainly not always 100% but I think it's safe to use one. Can help with duplicate content issues as well, for example a printer-friendly version of a page or archives. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - The Largest LucasArts Gaming Community on the Internet To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - tell me your comments! To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Your #1 place to discuss all things World of Warcraft.

04-03-2009, 05:47 PM	#8
Trexle Member Join Date: Jan 2009 Posts: 119	it guides the SE's what to index and what not to .. simple!!!! __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-28-2009, 07:47 PM	#3
rockinaway Apprentice Join Date: Oct 2006 Posts: 286	Yup, no problem at all. Although I would recommend you have one personally to control the spiders, but yeah, shouldn't be a problem.

03-29-2009, 07:56 PM	#5
rockinaway Apprentice Join Date: Oct 2006 Posts: 286	Strangely enough, I read something recently that said you shouldn't have a robots.txt file because the more areas you exclude from spiders, the less frequently they will visit. Is there truth to that?

03-29-2009, 08:19 PM	#7
rockinaway Apprentice Join Date: Oct 2006 Posts: 286	Damn, I can't find the actual link, but I am pretty sure I read it somewhere for SEO optimisation. It did shock me a bit, because I always thought it was better to use one.

04-03-2009, 08:10 PM	#9
Niche Member Join Date: Feb 2008 Posts: 62	I don't use robots file on most of my sites and I still get high position in the SERPs __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-04-2009, 05:32 PM	#10
Brandon Sheley Super Moderator Join Date: Nov 2005 Location: Topeka, KS Posts: 1,739	You don't "have" to have one, but it's strongly suggested unless every page on your site, you want crawled if it's just an html site, this might be okay, but any scripts or whatnot, I would add one. __________________ How to be To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. & Comparing To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. . Check out my blog , it's a DOFOLLOW blog, go comment! To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. Are you on Twitter? To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How to build a Helpful & Working robots.txt	Jolteon	Tutorials	13	03-19-2010 10:24 AM
Robots.txt And Your Forum	Jolteon	Articles	0	12-27-2006 06:17 PM