How can I tell if my development site is indexed?
Do a quick site search by typing “site:MyDevelopmentSite.com” in Google – if you get no results then wahey – Happy days. If however Google returns a load of pages from the development site, then not quite so marvellous.
Why is this a problem?
Well it’s a problem because Google may see the development site as duplicate content to the main domain and may penalise the site you want to rank because of this. The development site may even rank higher than your main domain for your keywords. It’s also a problem for the end user, because the information on the development site might be out of date – prices or addresses may have changed. It seems a waste to ensure fantastic search engine optimisation of your new site, but then partially waste it by letting Google see a duplicate copy.
How did it happen?
Simply by putting a link to the development site somewhere so the search engines can find the link – e.g. putting the link in a forum, on Twitter or a blog may all result in the website being found. I have also experienced an instance when the main site homepage has a miniscule link at the bottom of the page with the anchor text of “test” which links to the test site – resulting in the whole test site being indexed.
How can I fix it?
Easy peasy. First go to your robots.txt file on the development site and add the line:
disallow: /
This tells all search engines that you don’t want them to crawl or index your site. This is a good start but it doesn’t work instantly and we really want Google to remove your development site ASAP. The next stage is to get Google Webmaster tools access for the development site. Once you’ve verified yourself as the webmaster using the available options (e.g. uploading an html file to the development site root or by adding a meta tag to the development site homepage), you can then go to “Site Configuration > Crawler Access” and click on the “Remove URL” tab. Click the button for a “New Removal Request” and then type in your URL e.g. “MyDevelopmentSite.com” and hit “continue”. You need to tick the check box to say you have added the “disallow” line in the robots.txt file. Google then tells you “This will remove all instances of http://MyDevelopmentSite.com/ from Google search results” – Good news- hit “Submit” once more, The request then goes into a status of “pending” – check back later in the day to see once the status has changed to “Removed” and your development site should be gone!
Related posts:
