If you want a website that works for both people and search engine spiders like
GoogleBot, you have to spider check your work. It sounds obvious and simple, but
if you don’t spider check your work – how do you know it works?
Search engine optimization or SEO is an environment where humans have limited
visibility. There is definitely a limit on how much human eyeballs alone can see
in terms of how the GoogleBot sees your website without actually spider checking
your work on Google.
Search engine spiders like GoogleBot are robot software that crawl your
website for ranking. Google says compared to humans, “Bots access pattern is
completely different” – one of the greatest understatements in Google’s
Webmaster Guidelines!
People tend to assume Google rankings are much more automatic than they
actually are due to this difference between how humans view a website and how
robot search engine spiders view it for ranking.
If you’re an SEO and live and breathe Google’s algorithms, many of the
problems you deal with on a daily basis are related to bridging the gap between
what works for human visitors to your website and what works for search engine
spiders that bring traffic.
Search engine visitors are the most affordable way of getting free targeted
traffic to your website. However, one of the biggest SEO mistakes most people
make is assuming their web designer is also a search engine optimization
expert.
In most instances, you should not expect the person primarily concerned with
the look and feel of your website and its coding to also keep up with the latest
in Google’s algorithms that are constantly evolving and updating. While web
design and SEO are both at the heart of your website’s functionality, never
assume your web designer is also an SEO expert.
SEO is a different knowledge and software set than most web designers can
devote time to or should be expected to stay on top of. SEO is also a one strike
you are out environment. You can do a hundred things right, but get one
important element of SEO wrong and it can undo everything else you do.
The question that can often make this point clear is to ask your web
designer, “How do you spider check your work?”
Although Google’s Webmaster Guidelines provide tools and guidance on how to
check a website through a spider’s eye view – like looking at the site with a
Lynx text based browser or using the Googlebot tool in Google Webmaster Tools –
in fact, very few web developers have actually read and followed Google’s
guidelines.
This is why spider checking your website is so important to see if what you
thought you were communicating on the Internet is actually being seen and ranked
on Google. If you haven’t spider checked your website, you simply can’t tell if
it is working on search engines. Here are a few easy ways to do that.
Site:Search
First enter the following in the Google search box:
site:yourdomain.com
A site:search is the single most important diagnostic search on all of the
major search engines that tells you how that search engine views your website.
Don’t put a space after the colon in a site:search or you won’t get the right
results.
Because your root domain without any slashes after the domain name is the top
of your site’s hierarchy, you always want to see your root homepage as the top
result of a site:search on Google.
If you don’t see your homepage at the top of a site:search, there may be a
problem. Most of your ranking strength is focused in your homepage where the
majority of the external links to a site usually point. The lack of appearance
of your homepage at the top of a site:search on Google is one of the ways you
can see if your site is under a penalty or downgrade – although this is not
conclusive evidence of that fact alone.
Note carefully how your homepage displays with a site:search in Google’s and
Bing’s listings. 65 characters are displayed of your homepage title in blue text
at the top of your search engine listing, 150 characters of meta description
appear under it – or a snippet of text from your body text that matches the
keywords from a search request.
Click Your “Cached “Link
Next, on Google and Bing move your cursor to the right of the search engine
listing to make your “Cached” link appear. On Yahoo, the cached link appears
below the listing. Note the cached date – the last time the spider returned to
your site for ranking.
If you get the result “Your search – site:yourdomain.com – did not match any
documents,” that means either your site is not being crawled and indexed on
Google – or you’ve entered the domain incorrectly so check your spelling
carefully. It is also possible it has been removed from the index as a result of
a penalty, although most of Google’s penalties don’t result in this extreme an
action.
One thing you do want to look at is if Google is displaying the cached date
immediately, or sitting on the results for a while running spam tests before
publishing them. If Google is publishing the cache immediately, that’s a good
sign.
Click “Text-only version” of your Google cache
Click the “Text-only version” link in the upper right corner of Google’s
cache.
This strips the website down to the body text and image alternative text
associated with graphic images that Google sees for ranking. After clicking the
“Text-only version” link in the upper right corner of Google’s cache, many
websites have major portions of their site or even the whole site disappear –
meaning Google can’t see your content for rankings.
Clicking back and forth in Google’s cache between the “Full version” and
“Text-only version” of your website is how you spider check your work to see if
GoogleBot and humans are seeing the same thing.
Google only gives you rankings for keywords it sees on your website in the
“Text-only version” – unless you have links that show those keywords.
Unless you perform this test, you simply can’t tell if spiders are seeing
your site properly or not – and very often, they aren’t and the web developer
and site owner don’t know it. When Google can’t see important elements of your
site as humans do, the result is that you have been hidden rather than promoted
on the Internet.
Spider Check Your Keyword Densities
Next, enter a search you want to compete for, and find your search result on
Google. Once again, move your cursor to the right of the Google listing to make
the “Cached” link appear and click it. Now you will see exactly how Google sees
your keywords. They are highlighted in the cache.
To get an approximation of your site’s keyword densities (should be between
1-2% in most cases), copy and paste Google’s “Text-only version” into Microsoft
Word and get a word count of how many words Google sees in the body text and
image alternative text of the page. Now do a “Control find” for your keywords to
see how many times they are actually mentioned.
A 1% keyword density is your keywords appearing once in a hundred words – 2%
twice in a hundred words. You can compete for many searches with keyword
densities outside of this 1-2% average, but you may not compete across as broad
a range of searches as each keyword algorithm is very unique. Keep in mind your
rankings are also dependent upon your keywords appearing in the link text
pointing to your site which you can’t see looking at the webpage because it is
an offsite ranking factor.
Other important ways to spider check your work is with Google Webmaster Tools
that give you a wealth of diagnostics about how your site appears on Google, and
Google Analytics that shows your traffic and what keywords are actually bringing
visitors to your site.
While spider checking your work on Google takes less than thirty seconds
involving three simple steps; 1) Move your cursor to the right of your Google
listing to make the “Cached” link appear, 2) Click your “Cached” link, and 3)
Click the “Text-only version” link in the upper right corner, never assume your
webmaster has performed this vital test.
If after doing this test, important elements humans see on your webpage are
not visible to the GoogleBot, your content has been effectively hidden from the
Internet in terms of search ranking. You need to study Google’s Webmaster
Guidelines to diagnose what the problem is – and carefully follow them if you
want people to find you at the top of the rankings for a Google search.