When you change your website, add new pages or launch a new site how do the search engines find the content on your site?
Search engines have automated software called a spider, a crawler or a bot which visits your site and using the internal linking structure of your site to discover the pages on the site.
This Bot will come back to you website a regular intervals to discover more pages and look again at the pages that have already been visited so changes will be picked up by them.
There are a number of things that you can do to ensure that the pages on the site are visible to search engines bots:
Make sure that all pages on your site are optimised and you have considered your website’s visibility and SEO.
Review the internal linking structure of your site to ensure that all the pages of the site can be found by visitors and make sure that your site does not have a spider trap or pages which aren’t connected t your internal structure (orphaned pages)
Make sure that the robots.txt file stops the search engines from finding any items on the site that you won’t want them to see.
Make sure you have an up to date sitemap.xml file.
Using Google Webmaster tools can help you to identify any issues with the crawling of your website, as Kerry mentioned in her blog earlier this week.
New pages or changes to your site should then be picked up by the search engine bots to aid the crawling of the site.
The time this indexing process takes can vary and if you have introduced thousands of new pages the crawling of these can take some time. However, if you have considered all of the above items then your website will be found by Google.
Related posts:
