Development websites and Google
11th March 2010 by Emily Mace
An important tool in the testing of any website is the development site which allows you to test new tools or pages on your website before sending them live to ensure that things are working right before presenting them to your visitors or the search engines.
If you have a development site it’s important to know that it could be found by Google and consequently indexed. This could mean bad news for your website, as this will tell Google that there are two versions of your website and could lead to duplicate content issues, which is not what you want damaging the search engine optimisation of your website.
The quickest way to ensure that your development website is not indexed by Google is to either IP lock it, so that only specific IP addresses can view the site or block access to the site using a username and password that blocks anyone without the access permissions from seeing the stei – and this means Google as well.
However, this might not be possible so the below steps will help you to ensure that your development website is not crawled by Google.
Firstly ensure that your development site can have a standalone and unique robots.txt file and then create and upload the below robots.txt file
User-agent: *
Disallow: /
Once you have done this add a Meta Robots tag
These two steps should help the site not be indexed by Google. However, run a site: check on the development site address in Google (e.g. site:dev.yoursite.co.uk) to ensure that no content from the development site has been indexed. If this has already happened then you can remove the site from Google’s listings using Google Webmaster Tools. To do this add the development website address to your Webmaster Tools account and verify it. Once you have done this go into the “Site Configuration” section and select “Crawler Access” from the menu. Clicking on the “Remove URL” tab will give you the option to create a new removal request. This option allows you to ask Google to remove your entire site. Once you have selected this option Google will check to ensure you have blocked the whole site from being crawled using the Robots file and/or meta tag mentioned above, and once it’s confirmed that the Development site is supposed to be uncrawlable it will remove the listings for the site from Google. Obviously it’s important to make sure that you ONLY do this on the development site and NOT your live website!
If you follow the above guidelines you can enjoy the benefits of having a development website to test your changes without worrying that it will appear in Google’s listings.
Related Posts
- Why Is My Development Site Indexed?
- Tip for SEO Pro’s – e-commerce websites
- Google Introduce new Methods of Verifying websites in their Webmaster Tools
- Google page removal requests
- Getting more from Google Webmaster Tools
- Checking your Robots.txt file
- More on Robots.txt
- Google may be crawling Flash websites- but it’s not good for your SEO Yet!