Comments on: How to Prevent Pages Being Indexed by Search Engines

By: Kilpailut ja arvonnat – Linkkejä | Arpanappula

Kilpailut ja arvonnat – Linkkejä | Arpanappula — Mon, 16 Aug 2010 16:18:04 +0000

[...] http://newwebmasters.net/link-building/how-to-prevent-pages-being-indexed/ [...]

By: corbyboy

corbyboy — Thu, 18 Feb 2010 18:55:50 +0000

Spiders can still get through as it is only designed for honest spiders such as search engines.

There should only be a single robots.txt file in the root directory.

By default, domain.com/love.html will be indexed, but if you want domain.com/love/me.html not to be indexed you can use:

User-agent: *
Disallow: /love/me.html

By: Andrew

Andrew — Thu, 18 Feb 2010 18:08:33 +0000

Thanks! This helped me a lot. I just needed one page to not get indexed, but I needed my visitors to see it – so I opted for the ‘nofollow’ method.

The robots.txt thing is really confusing – although, everywhere else it seems to be the only option? It seems pretty useless if the spiders can still get through?

When you are talking about the user-agent in the robot.txt file, you mention putting it the websites main directory – wouldn’t that make the entire website prone to not getting indexed? How would you use that method for a specific webpage?

For example, I want ‘domain.com/love.html’indexed, but I don’t want ‘domain.com/love/me.html’ indexed … where would the robot.txt go? In the ‘Love’ Directory?

Thanks again!

By: F. Andy Seidl

F. Andy Seidl — Sun, 21 Sep 2008 03:12:25 +0000

As a webmaster, you definitely should use user-agent headers to manager server traffic. But understand that this is purely a pragmatic tactic and not a serious security measure.

I wrote more about this here:

Webmaster Tips: Blocking Selected User-Agents
http://faseidl.com/public/item/213126

By: Websites tagged "indexed" on Postsaver

Websites tagged "indexed" on Postsaver — Thu, 11 Sep 2008 05:47:22 +0000