Google Likes You to Help With Crawl Efficiency

Back in June, there was a big discussion about the whole page sculpting/nofollow thing. The subject came up again in August when Search Engine Roundtable highlighted something Matt Cutts said in the I/O conference about adding nofollow to login page links.

As Vanessa Fox pointed out in the comments, Google have several times mentioned that using nofollow on links that lead to low value pages (like login or registration pages) is a good and acceptable use. What caught my eye was that she pointed out that Googlebot will only spend so much resource on a website, so why give it any content that it can’t do anything with?

We have noticed this on several occasions – we check a site frequently for what we call site integrity – which includes things like broken links, crawling problems, and duplicate content. Checking on Googlebot activity and the pages that are indexed on the site is something that you can do to double check on any site crawling problems.

Google Webmaster Tools provides Crawl Stats graphs, which can be useful for diagnosing problems. Here is one from a client that had a new website that had a spider trap. You can see right at the beginning of this graph that the activity shows a spike of just short of 5000 pages crawled. This should raise alarm bells if you know that the site doesn’t actually have that many pages (this site had about 1000). Locating and fixing the spider trap was a priority, then getting Googlebot to come and revisit the site again resulted in the smaller spikes that you can see, after which the site began to rank for its keywords.

I talked the other day about XML sitemaps and that doing a crawl for these can show up spider traps so this is another way of detecting these kind of problems.

So back to the subject of Google’s crawl efficiency – Google definitely has a preference for sites that have a good number of quality pages compared to low quality pages. So closing spider traps, nofollowing links to login pages, excluding search results pages, content you have behind forms and drops downs and making sure that there isn’t an enquiry form for every single one of your products can be of real value to the ability of a site to rank well. Therefore helping Google with their crawl process helps your SEO.