Comments on: An In-Depth robots.txt Guide http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/ Making Money through Ethical Internet Marketing. Thu, 02 Feb 2012 02:50:34 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Udegbunam Chukwudi http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-12311 Udegbunam Chukwudi Mon, 17 Aug 2009 06:32:12 +0000 http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-12311 Inserting a blank html or a html file that automatically refreshes to your blog, in your download folder adds extra security against those who go snooping around at your robots.txt file. That's what I do at my site/blog. Inserting a blank html or a html file that automatically refreshes to your blog, in your download folder adds extra security against those who go snooping around at your robots.txt file. That’s what I do at my site/blog.

]]> By: Internal Linking - META nofollow, rel nofollow, robots.txt Confusion http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-11976 Internal Linking - META nofollow, rel nofollow, robots.txt Confusion Fri, 26 Jun 2009 18:34:00 +0000 http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-11976 [...] The robots.txt file is a simple txt file that you can create with any basic text editor like notepad, wordpad etc. It is uploaded into the root of a site and named robots.txt For a more in-depth explanation, refer to my robots.txt guide. [...] [...] The robots.txt file is a simple txt file that you can create with any basic text editor like notepad, wordpad etc. It is uploaded into the root of a site and named robots.txt For a more in-depth explanation, refer to my robots.txt guide. [...]

]]> By: Tamara http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-4244 Tamara Tue, 01 Apr 2008 21:43:49 +0000 http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-4244 Ok my website is not done yet....but you are too funny with your butt pictures. I am so glad you brought this fact out, very interesting! Very usefull information! Ok my website is not done yet….but you are too funny with your butt pictures.

I am so glad you brought this fact out, very interesting!

Very usefull information!

]]> By: Will the WordPress “All in One SEO Pack” Plugin Help or Hinder Your Blog? | SEONoobs http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3880 Will the WordPress “All in One SEO Pack” Plugin Help or Hinder Your Blog? | SEONoobs Sat, 08 Mar 2008 18:36:22 +0000 http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3880 [...] Spaulding provides an excellent explanation of the robots.txt file on his [...] [...] Spaulding provides an excellent explanation of the robots.txt file on his [...]

]]> By: Duplicate Content - It May Not be so Bad http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3547 Duplicate Content - It May Not be so Bad Wed, 13 Feb 2008 23:33:12 +0000 http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3547 [...] be indexed while the useless pages will not. To ensure the right pages are indexed be sure to use robots.txt and a good internal linking structure. If you’re running a WordPress blog there are many ways [...] [...] be indexed while the useless pages will not. To ensure the right pages are indexed be sure to use robots.txt and a good internal linking structure. If you’re running a WordPress blog there are many ways [...]

]]> By: Free Blog Review #3 - ThomasSinfield dot com | Internet Marketing Blog with Josh Spaulding http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3122 Free Blog Review #3 - ThomasSinfield dot com | Internet Marketing Blog with Josh Spaulding Mon, 21 Jan 2008 18:45:27 +0000 http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3122 [...] It wouldn’t hurt to restrict some directories etc. that don’t need to be crawled. Anyone can feel free to view my robots.txt and/or read my in-depth robots.txt guide. [...] [...] It wouldn’t hurt to restrict some directories etc. that don’t need to be crawled. Anyone can feel free to view my robots.txt and/or read my in-depth robots.txt guide. [...]

]]> By: Josh Spaulding http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3117 Josh Spaulding Mon, 21 Jan 2008 14:13:32 +0000 http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3117 Thanks Maurice, I've never heard of a robots.txt generator, but that sounds like a good option for those who may be intimidated by them, although I don't see a reason to be intimidated. The bad bot exclusion seems like a nice option. I'm not a techie, as you know, but I can't believe a "bad" bot would obey a robots.txt file though. They are "bad" aren't they? So why would they obey robots.txt? :) I have seen many files disallowing alot of bad bots though, so I guess it does work. Thanks Maurice,

I’ve never heard of a robots.txt generator, but that sounds like a good option for those who may be intimidated by them, although I don’t see a reason to be intimidated.

The bad bot exclusion seems like a nice option. I’m not a techie, as you know, but I can’t believe a “bad” bot would obey a robots.txt file though. They are “bad” aren’t they? So why would they obey robots.txt? :) I have seen many files disallowing alot of bad bots though, so I guess it does work.

]]> By: Best Posts for the Week of January 14th 2008 http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3106 Best Posts for the Week of January 14th 2008 Mon, 21 Jan 2008 04:57:43 +0000 http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3106 [...] An In-Depth robots.txt Guide. [...] [...] An In-Depth robots.txt Guide. [...]

]]> By: Maurice (TheCaymanHost) http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3083 Maurice (TheCaymanHost) Sun, 20 Jan 2008 00:52:02 +0000 http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3083 Good stuff Josh - a nicely written explanation. There are several free robots.txt generators available online - I prefer Yellowpipe.com which gives anyone a good starting point and it's as easy as filling in the blanks on a form. It also offers the option to exclude known "bad bots" which is useful. Either way, it's a nice simple way for people to build an error free robots.txt file, which is great for techno morons like me :0) Good stuff Josh – a nicely written explanation.

There are several free robots.txt generators available online – I prefer Yellowpipe.com which gives anyone a good starting point and it’s as easy as filling in the blanks on a form.

It also offers the option to exclude known “bad bots” which is useful. Either way, it’s a nice simple way for people to build an error free robots.txt file, which is great for techno morons like me :0)

]]> By: Stumble into the Weekend 01/18 http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3062 Stumble into the Weekend 01/18 Sat, 19 Jan 2008 01:38:33 +0000 http://www.ethicalim.com/search-engine-optimization/an-in-depth-robotstxt-guide/#comment-3062 [...] If you’re not sure what I’m talking about, just read Josh Spaulding’s “In-Depth robots.txt Guide“. Don’t neglect [...] [...] If you’re not sure what I’m talking about, just read Josh Spaulding’s “In-Depth robots.txt Guide“. Don’t neglect [...]

]]>