If you want a website that works for both people and search engine spiders like GoogleBot, you have to spider check your work. It sounds obvious and simple, but if you don’t spider check your work – how do you know it works?
Search engine optimization or SEO is an environment where humans have limited visibility. There is definitely a limit on how much human eyeballs alone can see in terms of how the GoogleBot sees your website without actually spider checking your work on Google.
Search engine spiders like GoogleBot are robot software that crawl your website for ranking. Google says compared to humans, “Bots access pattern is completely different” – one of the greatest understatements in Google’s Webmaster Guidelines!
People tend to assume Google rankings are much more automatic than they actually are due to this difference between how humans view a website and how robot search engine spiders view it for ranking.
If you’re an SEO and live and breathe Google’s algorithms, many of the problems you deal with on a daily basis are related to bridging the gap between what works for human visitors to your website and what works for search engine spiders that bring traffic.
Search engine visitors are the most affordable way of getting free targeted traffic to your website. However, one of the biggest SEO mistakes most people make is assuming their web designer is also a search engine optimization expert.
In most instances, you should not expect the person primarily concerned with the look and feel of your website and its coding to also keep up with the latest in Google’s algorithms that are constantly evolving and updating. While web design and SEO are both at the heart of your website’s functionality, never assume your web designer is also an SEO expert.
SEO is a different knowledge and software set than most web designers can devote time to or should be expected to stay on top of. SEO is also a one strike you are out environment. You can do a hundred things right, but get one important element of SEO wrong and it can undo everything else you do.
The question that can often make this point clear is to ask your web designer, “How do you spider check your work?”
Although Google’s Webmaster Guidelines provide tools and guidance on how to check a website through a spider’s eye view – like looking at the site with a Lynx text based browser or using the Googlebot tool in Google Webmaster Tools – in fact, very few web developers have actually read and followed Google’s guidelines.
This is why spider checking your website is so important to see if what you thought you were communicating on the Internet is actually being seen and ranked on Google. If you haven’t spider checked your website, you simply can’t tell if it is working on search engines. Here are a few easy ways to do that.
Site:Search
First enter the following in the Google search box:
site:yourdomain.com
A site:search is the single most important diagnostic search on all of the major search engines that tells you how that search engine views your website. Don’t put a space after the colon in a site:search or you won’t get the right results.
Because your root domain without any slashes after the domain name is the top of your site’s hierarchy, you always want to see your root homepage as the top result of a site:search on Google.
If you don’t see your homepage at the top of a site:search, there may be a problem. Most of your ranking strength is focused in your homepage where the majority of the external links to a site usually point. The lack of appearance of your homepage at the top of a site:search on Google is one of the ways you can see if your site is under a penalty or downgrade – although this is not conclusive evidence of that fact alone.
Note carefully how your homepage displays with a site:search in Google’s and Bing’s listings. 65 characters are displayed of your homepage title in blue text at the top of your search engine listing, 150 characters of meta description appear under it – or a snippet of text from your body text that matches the keywords from a search request.
Click Your “Cached “Link
Next, on Google and Bing move your cursor to the right of the search engine listing to make your “Cached” link appear. On Yahoo, the cached link appears below the listing. Note the cached date – the last time the spider returned to your site for ranking.
If you get the result “Your search – site:yourdomain.com – did not match any documents,” that means either your site is not being crawled and indexed on Google – or you’ve entered the domain incorrectly so check your spelling carefully. It is also possible it has been removed from the index as a result of a penalty, although most of Google’s penalties don’t result in this extreme an action.
One thing you do want to look at is if Google is displaying the cached date immediately, or sitting on the results for a while running spam tests before publishing them. If Google is publishing the cache immediately, that’s a good sign.
Click “Text-only version” of your Google cache
Click the “Text-only version” link in the upper right corner of Google’s cache.
This strips the website down to the body text and image alternative text associated with graphic images that Google sees for ranking. After clicking the “Text-only version” link in the upper right corner of Google’s cache, many websites have major portions of their site or even the whole site disappear – meaning Google can’t see your content for rankings.
Clicking back and forth in Google’s cache between the “Full version” and “Text-only version” of your website is how you spider check your work to see if GoogleBot and humans are seeing the same thing.
Google only gives you rankings for keywords it sees on your website in the “Text-only version” – unless you have links that show those keywords.
Unless you perform this test, you simply can’t tell if spiders are seeing your site properly or not – and very often, they aren’t and the web developer and site owner don’t know it. When Google can’t see important elements of your site as humans do, the result is that you have been hidden rather than promoted on the Internet.
Spider Check Your Keyword Densities
Next, enter a search you want to compete for, and find your search result on Google. Once again, move your cursor to the right of the Google listing to make the “Cached” link appear and click it. Now you will see exactly how Google sees your keywords. They are highlighted in the cache.
To get an approximation of your site’s keyword densities (should be between 1-2% in most cases), copy and paste Google’s “Text-only version” into Microsoft Word and get a word count of how many words Google sees in the body text and image alternative text of the page. Now do a “Control find” for your keywords to see how many times they are actually mentioned.
A 1% keyword density is your keywords appearing once in a hundred words – 2% twice in a hundred words. You can compete for many searches with keyword densities outside of this 1-2% average, but you may not compete across as broad a range of searches as each keyword algorithm is very unique. Keep in mind your rankings are also dependent upon your keywords appearing in the link text pointing to your site which you can’t see looking at the webpage because it is an offsite ranking factor.
Other important ways to spider check your work is with Google Webmaster Tools that give you a wealth of diagnostics about how your site appears on Google, and Google Analytics that shows your traffic and what keywords are actually bringing visitors to your site.
While spider checking your work on Google takes less than thirty seconds involving three simple steps; 1) Move your cursor to the right of your Google listing to make the “Cached” link appear, 2) Click your “Cached” link, and 3) Click the “Text-only version” link in the upper right corner, never assume your webmaster has performed this vital test.
If after doing this test, important elements humans see on your webpage are not visible to the GoogleBot, your content has been effectively hidden from the Internet in terms of search ranking. You need to study Google’s Webmaster Guidelines to diagnose what the problem is – and carefully follow them if you want people to find you at the top of the rankings for a Google search.
It seems that Google only return Sitelinks when they are confident that the search query is looking for a specific site.