Following on from Pete’s blog about Sitemap.xml files I thought I’d write some more about Robots.txt files.
A Robots.txt file is important for your website’s SEO as it makes sure that you are only showing Google and the other search engines the best bits of your website and not sending them down the route of having non important items showing in your search listings. These include things like the admin section of your website or your PDF brochure if this contains content which is duplicated on your main site.
Robots files are supported by Google, Bing and Yahoo however, I was reading an article the other day where it seems that some people are finding that Bing aren’t honouring the content of Robots files on some websites. I’ve looked at some of the website’s I work on and I’m not seeing this issue myself yet, but it’s something I’m going to keep an eye on.
Using Robots.txt I have had some great success sorting some crawling issues on client websites. Along with the canonical tag the Robots.txt file has allowed me to address some duplicate content issues and to remove elements of websites which could create spider traps from the listings of search engines.
When working on your website it’s good practice to review your Robots.txt file regularly to make sure that there aren’t any new items you need to add to it. Making a regular check of the site: listings for your site can help you review the pages Google can currently see to ensure that nothing you don’t want search engines to see is listed. This will help you ensure that the pages on your website that are ranking in search engines are the most relevant to your services.
Related posts:
