Google 404 Error Page

Yahoo 404 Error Page

Bing 404 Error Page

They also provide information on how to produce good 404 error pages. At the bottom of that article they provide a list of items to include in your 404 error page (read more about HTTP status codes). These are supposed to help the user find what they are looking for and to ensure they stay on your website.

In this article we are going to look at the 404 error pages on Google, Yahoo and Bing to see how many of these guidelines they follow themselves. The sites are given a mark out of 10 for how well they meet each guideline.

We are going to access the following URLs:

• http://www.google.com/fake-page
• http://www.yahoo.com/fake-page
• http://www.bing.com/fake-page

Tell visitors clearly that the page they’re looking for can’t be found. Use language that is friendly and inviting

Google

The page – www.google.com/fake-page – does not exist.

The message that the page cannot be found is pretty clear although the error text is the same size as the rest of the page.

Mark: 4/10

Yahoo

Sorry, the page you requested was not found.

This text is larger than the rest of the page, is in bold and is centered. It is very clear that the page cannot be found.

Mark: 8/10

Bing.com

Let’s try that again

Although there is some explanatory text below, this is a terrible header to use for an error page. It is very vague and really doesn’t help the user.

The text “That web page doesn’t exist. Let’s see if we can help you find what you are looking for” is much more useful. If they used this as the header they would definitely improve the page.

However, it is the only error page that actually used a <h1> tag for the page header.

Mark: 4/10

Make sure your 404 page uses the same look and feel (including navigation) as the rest of your site.

Google

The page features the standard Google search box and logo at the top and the familiar links to popular Google sites such as Videos, Maps and News.

Mark: 8/10

Yahoo

Features the Yahoo logo and the links back to the homepage and help sections. Doesn’t feature the standard links to Yahoo Mail that seem to feature on a lot of Yahoo pages. However, each section of Yahoo seems to have its own style and design so this would be a difficult rule for Yahoo to follow.

Mark: 5/10

Bing

Has the standard site header, log and search box at the top of the page. It is even customised for your location and whether you are signed in or not.

Mark: 10/10

Consider adding links to your most popular articles or posts, as well as a link to your site’s home page.

The way I read this is that Google is recommending that as well as linking to your homepage, you should direct users to some popular sections of your site that they might be looking for.

Google

The only link to the homepage is via the Google logo at the top. Aside from that, the only even remotely useful link is to the help center. And the help topics it links to are not even specific for the error we have encountered. Not very helpful at all.

Mark: 2/10

Yahoo

Yahoo features a link to the homepage via the logo at the top and a text link in the main body. There is also a link to a list of Yahoo’s services and Yahoo Help Central. Like Google, this help center doesn’t link to a specific page that is relevant to our current problem.

Mark: 3/10

Bing

Bing features their logo at the top which links to the homepage and a text link to the help pages. As with the other two, not a link to help us with our current problem.

Mark: 3/10

Make sure that your webserver returns an actual 404 HTTP status code when a missing page is requested.

All the pages returned proper 404 headers. It would have been terribble news if they hadn’t.

Mark: All sites 10/10

Use the Enhance 404 widget to embed a search box on your custom 404 page and provide users with useful information to help them find the information they need.

None of the search engines actually uses this widget, but what it is trying to achieve is very useful.

It provides a search box and tries to extract query data from the URL to put in this box. This then allows you to search the current site for what you are looking for.

It also tries to list some close match URLs (see an example).

Let’s see if the search engines have any of these features in their sites.

Google

Google has a search box, but it doesn’t just search Google, it is just Google’s standard web search.

There is no list of closest match URLs either. So say if I accidentally typed in this URL:

http://www.google.com/analatics

Google should be clever enough to know that I am really looking for Google analytics. But it doesn’t, and that’s very disappointing.

Mark: 0/10

Yahoo

Yahoo sufferes from the same problem as Google, no list of closest match pages and the search box doesn’t just search Yahoo.

Mark: 0/10

Bing

It seems this is standard practice on the big three search engines. No list of closest matches and no search box for just bing.com.

Mark: 0/10

Total Marks

Google

24/40 = 60%

Yahoo

26/40 = 65%

Bing

27/40 = 67.5%

You can see that each search engine has similar marks, and that they are fair, but could definitely be improved.

It seems that search engines could do with a dose of their own medicine before telling us how to design our own 404 pages.

Google supports the new tag

Last week, Google, Yahoo and Live all announced their support for a new header tag called the “canonical” tag. This is designed to let you tell the search engines an alternative URL to index for a page.

It is designed to remove duplicate content issues from your web pages by letting you specify the correct version of the page. It is like a 301 redirect, but is a “strong hint” rather than an order.

A good example might be if you use information from the query string to sort data on a page. Your website might generate the following pages

• http://example.com/cat1.php?sort=size&dir=desc
• http://example.com/cat1.php?sort=price
• http://example.com/cat1.php?sort=price&dir=asc
• http://example.com/cat1.php?sort=avail&dir=desc

Search engines would regard each of these as individual pages, even though they are essentially the same page. The information is just in a different order.

Allowing URLs like this could be detrimental to your SEO efforts as your PageRank would be split between the pages, and people could be linking to the different versions of each page.

You could use the tag like this to remove this problem. There is a more technical explanation of how to use the tag over at the SEOmoz blog.

<link rel="canonical" href="http://example.com" />

Lesson: DON’T Use This Tag

The examples given by Google underline something very important about this tag; you should be striving to never need to use it.

Here are some situations that you should never need to use the tag to resolve:

WWW vs Non-WWW URLs

In order to prevent duplicate content, a lot of websites redirect the non-www version of a URL to the www version. For example, accessing http://example.com will redirect you to http://www.example.com.

Lesson: use a 301 redirect

Fixing Sloppy Links Generated By Your CMS

The links generated by the CMS or online store in Google’s examples are the result of sloppy coding.

While accessing http://example.com/prod.php?p=fish and http://example.com/prod.php?p=fish&cat=animals will result in the same URL, these two links should NEVER be generated by a CMS. This is the result of sloppy coding. While using the canonical tag as a backup might be OK, you should never rely on it.

Lesson: Fix your coding errors first.

Remove Tracking And Affiliate Links From URLs

Search results are full of URLs with tracking IDs, such as those generated by Google Analytics URL Builder. Like this:

• http://newwebmasters.net/?utm_source=email&utm_medium=gmail&utm_term=web&utm_campaign=test1

But these links should be logged, stripped from the URL and the user redirected, preferably via a 301 redirect.

Remove Session Identifiers

We are all familiar with sessions IDs in URLs. They look something like this:

• http://www.example.com/index.php?PHPSESSID=a2sd433345gftr

These should never be used on publicly accessible URLs. If you use PHP, always choose the default cookie option for recording session identifiers. Using identifiers in the URL is less reliable and less secure.

Google also claims that they are able to detect session identifiers in URLs, so in theory this shouldn’t be a problem.

Lesson: Never use sessions identifiers in your publicly accessible URLs. Never.

As an exception, one thing this tag will be great for combatting is the number of printable versions of documents I come across via search engines. There is very rarely a link to access the full version of the page directly from the printable version. If all these versions of pages were eradicated from the search results, I would be very happy. People should really be tagging these as noindex, but if they are featuring in the search results, I guess this won’t happen.

In Summary

As usual, most of these ideas are just my personal thoughts. While the new tag shouldn’t be necessary on your site, it is another useful tool to have handy.

But before you use it, think about what is happening on your website in the first place to make you need to use it.

What Exactly is PageRank?

PageRank is an algorithm invented by the founders of Google that assigns an “importance” rating to every page that it indexes. It is on a logarithmic scale from 0-10. This means that each of the 11 ratings is 10 times higher than the previous one. For example, PR 4 is ten times higher than PR 3. PR 8 is 100 times higher than PR 6.

Essentially, pages with a higher PR are more important, or more specifically, more popular, than pages with a lower PR.

What Determines PageRank?

When Google crawls a hyperlink, it counts it as a “vote” for the destination page from the source page. By linking to that page, the source page is saying “I endorse this website. It is good.” Importantly, however, each vote is weighted, so all votes are not equal. What makes a vote worth more? A higher PR of the source page of course. So pages with a higher PR have more influence in boosting the PR of other pages with a lower PR. So it is not just the quantity of links that influences PR, it is the quality of those links too.

Is all this confusing? It can be if you are unfamiliar with the concept. Let’s look at it with an illustration.

Example PageRank illustration

In this illustration the coloured circles represent pages on different websites. The numbers inside represent the PageRank of that page. The arrow represents a hyperlink. You can see that site A has 6 inbound links with a PR of 3, while site B has only 4 inbound links and a PR of 4. This is because the sites that link to site B have higher PRs themselves. They are more authoratative in the eyes of Google, therefore their links are more important.

Why is There Such A Buzz About PageRank?

We use more than 200 signals, including our patented PageRank

]]> http://newwebmasters.net/link-building/all-you-ever-wanted-to-know-about-pagerank/feed/ 0 http://newwebmasters.net/search-engines/live-webmaster-tools-now-detects-malware/ http://newwebmasters.net/search-engines/live-webmaster-tools-now-detects-malware/#comments Sat, 29 Nov 2008 13:10:25 +0000 corbyboy http://newwebmasters.net/?p=412

MSN’s Webmaster Tools Now detects malware

The latest update to MSN/Live’s Webmaster Tools notifies webmasters of any pages on their site that contains malware. The service is still playing catchup to Google Webmaster Central, but it is making progress.

The tool will also notifiy you if you are linking to any pages that contain malware, so you can remove the link and prevent your site being associated with them.

]]> http://newwebmasters.net/search-engines/live-webmaster-tools-now-detects-malware/feed/ 0 http://newwebmasters.net/link-building/google-guide-to-seo/ http://newwebmasters.net/link-building/google-guide-to-seo/#comments Thu, 13 Nov 2008 13:10:30 +0000 corbyboy http://newwebmasters.net/?p=380 Google has released what it calls “Google’s Search Engine Optimization Starter Guide” (pdf) over on the Official Google Webmaster Central blog.

After reading through it I have to say that it is an excellent introductory guide to the basics of SEO. It would very firmly go on my new webmaster reading list. It is clearly designed for the beginner, but as an experienced web developer even I appreciate a run through of the basics every now and again to refresh me. It runs through all of the advice using a fictional baseball card collector’s website as an example.

Some of the key advice given in the document includes:

• Create unique, accurate page titles
• Make use of the “description” meta tag
• Improve the structure of your URLs
• Make your site easier to navigate
• Offer quality content and services
• Write better anchor text
• Use heading tags appropriately
• Optimize your use of images
• Make effective use of robots.txt
• Be aware of rel=”nofollow” for links
• Promote your website in the right ways
• Make use of free webmaster tools
• Take advantage of web analytics services

All of these topics are wrapped up nicely into a 22 page pdf file.

Reading More Into Google’s Advice

Despite being aimed at beginners, there are a few little nuggets of information that the more experienced developer might be interested in.

The first that caught my eye was Google advice to “take advantage of” an image’s filename. I have never really thought of how Google indexes images, but it seems from this advice that the filename plays at least some part in this.

There is also confirmation of things we already suspected about Google, but weren’t completely certain, such as Google love affair with the blog (“Blog about new content or services”) and how important it is to have your site registered for local search (“Add your business to Google’s Local Business Center”).

So the guide is not just useful for the beginner, it is a recommended read for everybody.

]]> http://newwebmasters.net/link-building/google-guide-to-seo/feed/ 1 http://newwebmasters.net/link-building/fix-broken-links-with-google-webmaster-tools/ http://newwebmasters.net/link-building/fix-broken-links-with-google-webmaster-tools/#comments Tue, 14 Oct 2008 15:38:30 +0000 corbyboy http://newwebmasters.net/?p=358 Google Webmaster Tools has just released a great new feature that allows you to find out which sites are linking to non-existant (i.e. 404) pages on your website.

Take a look at the example below for this website:

Google Webmaster Tools Error Details

You can find the information in Webmaster Tools by clicking on Diagnostics > Web Crawl > Not Found. This screen will tell you how many people link to each non-existant page. If you click the figure, it will list all the URLs that link to each page. As with a lot of the data available in Webmaster Tools, you can also download it.

Redirecting from these non-existant URLs to ones that do exist is a great way of boosting the number of inbound links to your website. Links that point to non-existant pages will not help you rank in the search engines. Using a 301 redirect to point these links to a real page will help your rankings. Matt Cutts describes these as “free links to your site.”

Previously, it was very difficult to find the sites that linked to non-existant pages. This is a great addition to the webmaster tool collection and will become a very valuable resource for webmasters to increase the number of links to their site.

]]> http://newwebmasters.net/link-building/fix-broken-links-with-google-webmaster-tools/feed/ 1 http://newwebmasters.net/optimise/the-ultimate-guide-to-yahoo-search-operators/ http://newwebmasters.net/optimise/the-ultimate-guide-to-yahoo-search-operators/#comments Fri, 05 Sep 2008 23:24:07 +0000 corbyboy http://newwebmasters.net/?p=279

Advanced Yahoo Search Operators

Search engines are incredibly advanced pieces of software. The basic interface that we see and the speed at which they work often makes us forget that. Since search engine databases hold such vast quantities of information that they put in place special features that allow you to refine what you search for. These “operators” are very useful when optimising your website for the search engines.

Yahoo is probably the most popular place to get accurate information on which websites link where. The information on Google is very unreliable, as is MSN/Live. Yahoo’s Site Explorer also gives us a goldmine of useful linking information.

This article will help you to use the advanced search operators in Yahoo Search to optimise and analyse the links to and from your website as well as those to and from your competitors websites.

Why Do I Need To Analyse My Competitors’ Links?

If you find a useful, relevant and popular website that links to one of your competitors, you need to ask yourself “why is this site not linking to me?“

It is also useful to find out which “power” sites are linking to you. Sites such .edu and .gov sites are widely regarded as more important to search engines and it’s easy to find where these sites link to. (Note: it is not the actual .edu and .gov extensions that make a site more authorative. Sites like these simply tend to among the more established and linked-to sites on the Web) Perhaps you want to find out which .info sites are linking to you with spammy keywords.

Important note. When Yahoo returns results in its regular listings and in Site Explorer it lists nofollowed links along with followed links and doesn’t differentiate between them. You will have to check the links manually or use a special tool to check if the links are followed or not.

What Operators Can I Use?

Let’s get started on the actual operators themselves.

Link:

Use link: to find documents that link to a particular URL. Link: requires you to use a full domain including http://.

Example: link:http://search.example.com/

This operator will find any page that links the specified page. The example above will match http://search.example.com only. It will not match links to http://www.example.com, http://search.example.com/blog or http://help.search.example.com.

When you use the link: operator you need to be careful when choosing to use www or not in the domain. If you look for links to http://example.com, a page that links to http://www.example.com will not be returned unless the webmaster has set up a redirect, which many of them do.

Linkdomain:

Use linkdomain: to find pages that link to any URL on that domain and on any subdomains.

Example: linkdomain:example.com

This will match pages that link to http://example.com, http://www.example.com, http://search.example.com and http://search.example.com/blog.

Important note. When you do a search using link: or linkdomain: you will be redirected to Yahoo Site Explorer. Although this is a great tool, often you don’t want to be redirected. To avoid this happening, simply add one of the other search operators or some search text to your query.

Site:

Use site: your search to a particular domain and all its subdomains. Whereas link: and linkdomain: were looking for links to a site, the site: operator is conducting a search within a particular domain. For this reason, you need to use additional operators to your query or add some search text. If you just use site:example.com by itself you will be redirected to Yahoo Site Explorer.

Example: site:example.com horse.

This will search the domain example.com for the phrase “horse.” It will also match results on search.example.com and help.search.example.com.

Example: site: search.example.com horse.

This will match results on search.example.com and help.search.example.com but NOT example.com.

Hostname:

Use hostname: to find all documents from a particular host. It won’t search the subdomains of this host.

Example: hostname:example.com horse.

This will find results for the query “horse” from example.com. It won’t search blog.example.com.

Example: hostname: search.example.com horse.

This will only find results from that particular subdomain. It won’t find results from example.com or help.search.example.com.

Url:

Use url: to find a specific document in Yahoo’s search index. This operator does not search a domain, it simply locates the page and provides a link to it. It is useful to determine if a specific page is indexed.

Example: url:http://www.example.com.

The full URL is required. In the above example url:www.example.com would not match any results. With this operator you need to be careful about whether you include www or not. This is one of the reasons why webmasters are encouraged to make a site with or without www resolve to the same page.

Inurl:

Use inurl: to search the actual URLs in the search index for the search terms.

Example: inurl:horse.

The example above will return any page with the term “horse” in the URL. The search term will be highlighted in the results. You can use inurl: in isolation without any other operators or search terms.

Intitle:

This operator will search the page titles for the search terms.

Example: intitle:horse.

The example above will return any page with the term “horse” in the http://newwebmasters.net/history/a-look-back-at-googles-acquisitions/ http://newwebmasters.net/history/a-look-back-at-googles-acquisitions/#comments Tue, 19 Aug 2008 09:19:50 +0000 corbyboy http://newwebmasters.net/?p=43 Over the years Google has acquired many companies. Some of the most innovative technolgoy products of the last few years have been snapped up by Google. Many of these they improve or incorporate into an exisiting product. Others, however, are left to stagnate or eventually are wound up.

According to the article on Wikipedia, Google has acquired 51 companies as of March 2008. The list below outlines most of them to give you an idea of just how much technology Google actually owns. The list is not comprehensive, but it lists all the big ones.

February 2001 – Deja

Deja was a Usenet discussion service that Google bought to develop into Google Groups. At the time Deja’s archive dated back to 1995. After the acquisition, the archives were extended back to 1981.

February 2003 – Pyra Labs Blogger Software

Blogger was first made available to the public in August 1999. Initially the product was offered for use free of charge but the company ran into financial trouble. After the loss of many employees, the company received some funding and released new ad-supported versions of Blogger. The company was acquired by Google in 2003, upon which the “pro” features were made available to everybody free of charge. Opening up features for free became a hallmark of Google’s acquisition policy.

April 2003 – Applied Semantics Online Advertising

The technology behind Google AdSense and AdWords is based on WordNet, a system that groups synonyms into sets based on their semantic relationships. A company called Oingo, a small search engine company founded in 1998, used this technology. Oingo changed its name to Applied Semantics in 2001. Google paid $102 million to acquire the company in 2003.

September 2003 – Kaltix Corp Search Engine

In the press release confirming the acquisition, Google said that it and Kaltix Corp “share a common commitment to developing innovative search technologies.” Google acquired that technology on 30 September 2003 and used it to develop iGoogle.

June 2004 – Baidu Chinese Search Engine – 2.6% Share

Estimates for the cost of the deal vary from $5 million to $10 million, but on 23 June 2004, Google bought a 2.6% share of Baidu, a Chinese language search engine. The money was used to upgrade their technology and to stabilise the company before it was floated on the Stock Market. In December 2007 Baidu became the first Chinese company to be included in the US NASDAQ100 Index.

July 2004 – Picasa Image Management Software

Picasa logo

Picasa became another service offered for free after Google’s acquisition from Idealab on 13 July 2004. Picasa has since become heavily integrated with Blogger.

September and October 2004 – Google Maps Enhancements

Between September and October 2004, Google purchased three companies: ZipDash, Where2 and Keyhole, Inc. The technology was used to enhance Google Maps and Google Earth, including Ride Finder.

March 2005 – Urchin Software Corporation

Google announced that they were purchasing Urchin on 28 March 2005. The company was founded in December 2005 and was originally called Web Depot. After a few years of serious investment they eventually became Urchin Software Corporation. Google used their Urchin web analytics software to develop Google Analytics.

August 2005 – Android Mobile Phone Software

Google and the Open Handset Alliance developed the Android software platform for mobile devices. It is based on the Linux operating system and allows users to develop using a java-like language.

The platform was announced on 5 November 2007 by the Open Handset Alliance, which consists of 34 different companies.

December 2005 – AOL (5% Share)

AOL Logo

On 20 December Google announced that they were paying $1 billion for a 5% stake in AOL. The investment allowed the two companies to co-operate on search and advertising. Prior to the acquisition the two companies had worked together on other projects.

January 2006 – dMarc Broadcasting

dMarc Broadcasting, Inc. is an advertising company that specialises in radio advertising. Upon purchase, Google planned to integrate the technology into the AdWords platform. Google paid an initial $102 million for the company.They also announced that they full purchase price could rise to $1.136 billion over three years based on performance and revenue.

February 2006 – MeasureMap

MeasureMap was a small company that developed statistics and analytics software specifically for blogs. Google acquired them on 14 February 2006. The software was incorporated into Google Analytics.

March 2006 – Upstartle

Upstartle’s online word processor Writely formed the basis for Google Docs. The acquisition was announced on 9 March 2006. Registrations were closed for a few months as the service was integrated. Google Docs now includes spreadsheet and presentation functions.

June 2006 – 2Web Technologies

Most of the Google spreadsheet software was developed by Google itself. However, on 1 June 2006 it quietly bought a company called 2Web Technologies and integrated their flagship software XL2Web into the program. As with other software, the features became free to use once it was in Google’s hands.

October 2006 – YouTube

YouTube logo

In what was probably its most famous acquisition, Google purchased video sharing site YouTube for $1.65 billion in stock. The two brands remain distinct and Google own “Google Video” remains in place. The statistics for YouTube are simply amazing. Although not released very often, the company did state that in 2006, 50,000 videos were being added every day.

October 2006 – JotSpot

Google acquired JotSpot on 31 October 2006. It was rebranded as Google Sites and the features became free to use. It is currently still in beta.

February 2007 – AdScpae

Google acquired AdScape on 16 February 2007. It is a small company that specialises in in-game advertising. Google stated about the deal “In-game advertising is an area where we believe Google could add a lot of value to users.” Google reportedly paid $23 million for the company.

April 2007 – DoubleClick

DoubleClick Logo

In its most expensive acquisition, Google purchased DoubleClick, a huge player in the online advertising industry, for $3.1 billion in cash. It took until 11 March 2008 for the deal to be officially closed, following an antirust hearing. Soon after the acquisition, Google announced that they would be
cutting 300 jobs from the company.

June 2007 – Feedburner

Google paid a rumoured $100 million for Feedburner on 3 June 2007. The next month the PRO features were made available for free. The company provides services for web feeds.

October 2007 – Jaiku

Jaiku Logo

Jaiku is a micro-blogging software company founded in Helsinki, Finland. It is like Twitter, but for your mobile phone. It is also more fully-featured than Twitter. New registrations remain closed and news on Google’s future plans for the platform are quite scarce.

]]> http://newwebmasters.net/history/a-look-back-at-googles-acquisitions/feed/ 0 http://newwebmasters.net/link-building/exploring-msns-new-webmaster-tools/ http://newwebmasters.net/link-building/exploring-msns-new-webmaster-tools/#comments Tue, 12 Aug 2008 10:38:44 +0000 corbyboy http://newwebmasters.net/?p=203

Live Search keeps improving all the time

MSN has recently upgraded their Webmaster Center to provide additional tools. Previously they provided basically no information but now they provide some excellent tools and some great nuggets of information. We will look at them in this article.

Websites Overview

Overview of Live Webmaster Center

The first page you arrive at will list all the websites that you have previously registered. You can see the authentication code that must be present to use your site at Live Webmaster Center. You can also see whether you have verified the site by XML tag or by meta tag.

From this page you can also add a new website by clicking “Add a site.” Click the URL of the site you want to access.

Website Summary

Summary page for your chosen website

The next page gives the summary of all the information that Live has about your website. Some excellent statistics are listed here. You can see when was the last time the Live crawler accessed your website. You can also see how many pages are indexed and whether your website is blocked or not.

Live now lists an interesting metric called “Domain score.” This is how authoratative Live regards your website to be. It is rated from 0 to 5.

Underneath Site Status you can see the top five pages on your website. Presumably these are ordered by Page score. You can see the last time Live accessed each of these pages and whether they are blocked.

Website Profile

Website profile page

On this page you can give Live the address of your website sitemap. You can change how your website is verified. You can also add an email address so that you can be contacted if there are any crawl issues with your website.

Crawl Issues

View crawl issues with your website

On this page you can see any issues that were encountered when Live crawled your website. One of the most useful parts of this section is that you can filter the errors by subdomain or folder. According to Live’s help files filters can only be applied when 1,000 or more errors are found. In practice this doesn’t seem to be the case however.

You can lists pages that were not found (404 errors), pages that were blocked by robots.txt, pages that have long dynamic URLs and unsupported content types. The first 20 results will be listed here but you can download the first 1,000 in csv format.

Backlinks

Website backlink data

The backlink feature lists all the incoming links to your website. Only the first 20 results are displayed, but you can download the first 1,000.

Filtering to get Useful Data

The filtering tools in this section are very useful. First of all you can exclude all internal links. Type in your website domain into the filter box. Check the filter radio button and choose the “Exclude” option. This will now list all the external links to your website.

You can also filter the results by tld. For example you can list list all the .edu domains that link to your website. Check the filter radio button and type “.edu” into the filter box. Choose to “include” this extension.

Outbound Links

View outbound link data

This page lists all the websites that your site links to. As before only 20 pages are listed but you can download 1,000. These results can be filtered in the same way that the backlink data can.

Keywords Tool

View keyword performance

This is a mysterious tool that shows in which order pages are returned for a certain search term. The results do not give any indication at which number the pages appear in the search results, just in which order pages from your site are returned. The page score and last crawled date are also returned for each page. For the time being the real usefulness of this page is questionable.

After the lastest update, Live Webmaster Tools certainly is much more useful. It need some improvement, particularly in the keywords search section. The algorithm to determine page score also needs tweaking. Too many pages are being returned with page score five. Overall though, an excellent improvement.

]]> http://newwebmasters.net/link-building/exploring-msns-new-webmaster-tools/feed/ 0 http://newwebmasters.net/link-building/how-to-prevent-pages-being-indexed/ http://newwebmasters.net/link-building/how-to-prevent-pages-being-indexed/#comments Sat, 09 Aug 2008 23:01:46 +0000 corbyboy http://newwebmasters.net/?p=196

Keep those robots at bay

As much as webmasters are desperate for search engine spiders to crawl their pages, there are often times when you want a page or section to be ignored. They might be non-content pages such as terms and conditions or a privacy policy. They could be test pages or just private pages that you don’t want to share.

This article will show you all the different ways to keep search engines away from your content. The most appropriate method to use for your own situation is up to you.

Robots.txt

The standard way of preventing search engines accessing a page is to block it using a robots.txt file. It goes in the the base directory of your website.

You use the following syntax:

User-agent: *
Disallow: /sessions/
Disallow: /cgi-bin/

The * under User-agent means that the rules apply to all spiders. The Disallow rule lists the directories that should be excluded.

You can do other things with robots.txt:

User-agent: Googlebot
Disallow:

User-agent: *
Disallow: /

These rules in this example mean that Googlebot is allowed to access the entire site (note how there is nothing after Disallow). Every other user-agent is blocked. There are more advanced examples of how to build a robots.txt file at robotstxt.org.

There are two important things to remember about using robots.txt. The first is that a spider can ignore it. This is particularly true for robots harvesting email addresses or scraping your content. The second is that anybody can read the file. Don’t use robots.txt to hide secret information, as a quick glance at the file will tell you where all those secret directories are.

You should also be aware that even though search engines won’t spider the site, the URL can still appear in search engine result lists, especially if the page is linked to from another site.

META Tags

You can use meta tags to control how spiders index specific pages. They look something like this:

<meta name="robots" content="noindex,nofollow" />

The options that you have concerning robots are:

• “index” – which tells the robot to index that page
• “noindex” – tells the robot to not index that page
• “follow” – the robot should follow all the links on that page
• “nofollow” – the robot should not spider any links from that page

The default is “index,follow” so if you want your page spidered and the links followed then you can omit the tag.

When using a meta tag to “noindex” a page you will not have the problem of the URL appearing in the search engine results. It will be completely omitted. Remember that a page that is not indexed will still be crawled and unless you have also specified “nofollow” in your tag, the links on that page will be spidered too.

Nofollow Links

You can use “nofollow” on a specific hyperlink to stop search engine spiders from following that link and crawling the page. It is supported by Google, Yahoo! and MSN/Live You use the following syntax:

<a href="http://example.com/page.html" rel="nofollow">Example.com</a>

This syntax was originally created due to the problem of comment spam on guestbooks and blogs. The original idea was that a site could be linked to without passing any PageRank or link influence.

While this method is useful to prevent a spider following a link to a page, remember that it only affects the specific hyperlink it is attached to. If another link without nofollow points to the same page then that link will be followed and the page will still be indexed.

Password protected pages

A common way to keep private pages secure is to protect them with a password. Since the page cannot be accessed without the proper credentials, search engine spiders cannot index the page. As with some of the other examples, the URL may still appear in search engine results if a link to the page is found.

The following code shows you how to protect a page with a username and password using PHP. It uses basic HTTP Authentication:

<?php
if (!isset($_SERVER['PHP_AUTH_USER'])) {
    header('WWW-Authenticate: Basic realm="My Realm"');
    header('HTTP/1.0 401 Unauthorized');
    echo 'Text to send if user hits Cancel button';
    exit;
} else {
    echo "Hello {$_SERVER['PHP_AUTH_USER']}.";
    echo "You entered {$_SERVER['PHP_AUTH_PW']} as your password.";
}
?>

To read more about authentication in PHP visit php.net

Offline Pages

It may seem simple but keeping your web pages offline is the perfect way to prevent them being indexed. Maybe they can be put on an intranet rather than fully online. If collaboration or viewing over the Internet is essential then this method isn’t suitable, but on many occasions it is.

Remove URLs From Search Index

Google and Yahoo! both have tools that allow you to remove specific URLs from their search index relatively quickly.

Google Webmaster Tools

If you have registered with Google Webmaster Tools then you can use this to remove a URL.

1. From the dashboard click the website that contains the URL.
2. Then click “Tools”, followed by “Remove URLs.” Google will give you some tips about getting your URL removed and keeping it like that.
3. The URL you want to remove should be blocked from further spidering. If you don’t ensure this then the page will simply be indexed again at some point in the future. Once you are certain this is the case, click “New Removal Request.”
4. You then choose what you want to remove. You can remove and individual URL, an entire directory, a whole site or the Google cached copy of a page. Make your selection and click “Next.”
5. Enter the URL to be removed and click “Submit Removal Request.”

Yahoo! Site Explorer

To use Yahoo! Site Explorer you need to verify your site.

1. From the Site Explorer homepage either search for the URL you want to remove or click “Explore” and locate it manually.
2. Click [Delete URL/Path] next to the URL you wish to remove. Any URLs below that particular folder will also be deleted.
3. You will be presented with a confirmation page which will list all pages to be removed. You can edit this list to keep certain URLs. Click “Update” to create a new list.
4. Click “Yes” to confirm your deletions.

A note on removing URLs with webmaster tools. Using Google and Yahoo’s webmaster tools requires you to verify your site. The deletions will only stay in effect as long as your site is verified. If at any point your site becomes unverified, your deleted URLs will return to the index.

Another important note. When you remove a URL, make sure it is blocked from further crawling via one of the other methods listed above. Otherwise your deleted URLs will be reindexed soon after they are removed.

]]> http://newwebmasters.net/link-building/how-to-prevent-pages-being-indexed/feed/ 6