Google is always trying to improve the relevance of its search results and to do that, they have to tweak their algorithm (or at least threaten to change it). It looks like the current focus is with web pages that are nothing more than search results themselves.
For example, if you search in Google for “plasma televisions” you’ll notice that several of the first page listings are actually sets of search results (e.g. Amazon and BizRate).
This is something that Matt Cutts mentions today in his blog and there also appears a new clause in the Google webmaster guidelines:
- Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.
So the first step by Google for dealing with this issue is to ask webmasters to add any search results pages/directories to their robots.txt file.
But as this issue has been added to their technical guidelines, it means that it is something that they are concerned about and may consider it to be spam (note that it has not been added to the quality guidelines just yet). So who knows – perhaps Google is also working on algorithm filters to identify and penalise sites that do not adhere to these new rules that will be implemented at some point in the future.
Until the introduction of Google’s site maps, this would have been a real problem. The search engine spiders do not fill out forms or press buttons and so providing a method to “browse” the site’s database was a necessary element of creating an “indexable” site. But using site-maps and other Google properties such as Google Base will now be the preferred method to get the full set of products into the Google index.
There have been questions raised recently about the “weight” of pages found naturally (through links) versus pages added via Site Maps – as site map entries do not gain the benefit of a independent link and the potential PageRank boost, anchor text relevance, and other related weighting elements provided by a natural link.. but this is still being debated and so for now, it seems that the best way forward with Google is to do both. Use site maps to ensure that all of your content is in the index and then ensure that you continue to build links to these pages so that they have relevance and meaning.
Related posts:
