How GoogleBot see's your website

If you want a website that works for both people and search engine spiders like GoogleBot, you have to spider check your work. It sounds obvious and simple, but if you don’t spider check your work – how do you know it works?

Search engine optimization or SEO is an environment where humans have limited visibility. There is definitely a limit on how much human eyeballs alone can see in terms of how the GoogleBot sees your website without actually spider checking your work on Google.
Search engine spiders like GoogleBot are robot software that crawl your website for ranking. Google says compared to humans, “Bots access pattern is completely different” – one of the greatest understatements in Google’s Webmaster Guidelines!
People tend to assume Google rankings are much more automatic than they actually are due to this difference between how humans view a website and how robot search engine spiders view it for ranking.
If you’re an SEO and live and breathe Google’s algorithms, many of the problems you deal with on a daily basis are related to bridging the gap between what works for human visitors to your website and what works for search engine spiders that bring traffic.
Search engine visitors are the most affordable way of getting free targeted traffic to your website. However, one of the biggest SEO mistakes most people make is assuming their web designer is also a search engine optimization expert.
In most instances, you should not expect the person primarily concerned with the look and feel of your website and its coding to also keep up with the latest in Google’s algorithms that are constantly evolving and updating. While web design and SEO are both at the heart of your website’s functionality, never assume your web designer is also an SEO expert.
SEO is a different knowledge and software set than most web designers can devote time to or should be expected to stay on top of. SEO is also a one strike you are out environment. You can do a hundred things right, but get one important element of SEO wrong and it can undo everything else you do.
The question that can often make this point clear is to ask your web designer, “How do you spider check your work?”
Although Google’s Webmaster Guidelines provide tools and guidance on how to check a website through a spider’s eye view – like looking at the site with a Lynx text based browser or using the Googlebot tool in Google Webmaster Tools – in fact, very few web developers have actually read and followed Google’s guidelines.
This is why spider checking your website is so important to see if what you thought you were communicating on the Internet is actually being seen and ranked on Google. If you haven’t spider checked your website, you simply can’t tell if it is working on search engines. Here are a few easy ways to do that.
Site:Search
First enter the following in the Google search box:
site:yourdomain.com
A site:search is the single most important diagnostic search on all of the major search engines that tells you how that search engine views your website. Don’t put a space after the colon in a site:search or you won’t get the right results.
Because your root domain without any slashes after the domain name is the top of your site’s hierarchy, you always want to see your root homepage as the top result of a site:search on Google.
If you don’t see your homepage at the top of a site:search, there may be a problem. Most of your ranking strength is focused in your homepage where the majority of the external links to a site usually point. The lack of appearance of your homepage at the top of a site:search on Google is one of the ways you can see if your site is under a penalty or downgrade – although this is not conclusive evidence of that fact alone.
Note carefully how your homepage displays with a site:search in Google’s and Bing’s listings. 65 characters are displayed of your homepage title in blue text at the top of your search engine listing, 150 characters of meta description appear under it – or a snippet of text from your body text that matches the keywords from a search request.
Click Your “Cached “Link
Next, on Google and Bing move your cursor to the right of the search engine listing to make your “Cached” link appear. On Yahoo, the cached link appears below the listing. Note the cached date – the last time the spider returned to your site for ranking.
If you get the result “Your search – site:yourdomain.com – did not match any documents,” that means either your site is not being crawled and indexed on Google – or you’ve entered the domain incorrectly so check your spelling carefully. It is also possible it has been removed from the index as a result of a penalty, although most of Google’s penalties don’t result in this extreme an action.
One thing you do want to look at is if Google is displaying the cached date immediately, or sitting on the results for a while running spam tests before publishing them. If Google is publishing the cache immediately, that’s a good sign.
Click “Text-only version” of your Google cache
Click the “Text-only version” link in the upper right corner of Google’s cache.
This strips the website down to the body text and image alternative text associated with graphic images that Google sees for ranking. After clicking the “Text-only version” link in the upper right corner of Google’s cache, many websites have major portions of their site or even the whole site disappear – meaning Google can’t see your content for rankings.
Clicking back and forth in Google’s cache between the “Full version” and “Text-only version” of your website is how you spider check your work to see if GoogleBot and humans are seeing the same thing.
Google only gives you rankings for keywords it sees on your website in the “Text-only version” – unless you have links that show those keywords.
Unless you perform this test, you simply can’t tell if spiders are seeing your site properly or not – and very often, they aren’t and the web developer and site owner don’t know it. When Google can’t see important elements of your site as humans do, the result is that you have been hidden rather than promoted on the Internet.
Spider Check Your Keyword Densities
Next, enter a search you want to compete for, and find your search result on Google. Once again, move your cursor to the right of the Google listing to make the “Cached” link appear and click it. Now you will see exactly how Google sees your keywords. They are highlighted in the cache.
To get an approximation of your site’s keyword densities (should be between 1-2% in most cases), copy and paste Google’s “Text-only version” into Microsoft Word and get a word count of how many words Google sees in the body text and image alternative text of the page. Now do a “Control find” for your keywords to see how many times they are actually mentioned.
A 1% keyword density is your keywords appearing once in a hundred words – 2% twice in a hundred words. You can compete for many searches with keyword densities outside of this 1-2% average, but you may not compete across as broad a range of searches as each keyword algorithm is very unique. Keep in mind your rankings are also dependent upon your keywords appearing in the link text pointing to your site which you can’t see looking at the webpage because it is an offsite ranking factor.
Other important ways to spider check your work is with Google Webmaster Tools that give you a wealth of diagnostics about how your site appears on Google, and Google Analytics that shows your traffic and what keywords are actually bringing visitors to your site.
While spider checking your work on Google takes less than thirty seconds involving three simple steps; 1) Move your cursor to the right of your Google listing to make the “Cached” link appear, 2) Click your “Cached” link, and 3) Click the “Text-only version” link in the upper right corner, never assume your webmaster has performed this vital test.
If after doing this test, important elements humans see on your webpage are not visible to the GoogleBot, your content has been effectively hidden from the Internet in terms of search ranking. You need to study Google’s Webmaster Guidelines to diagnose what the problem is – and carefully follow them if you want people to find you at the top of the rankings for a Google search.

What do you expect from your SEO Company?

Those of us in the SEO industry operate by some pretty simple principles. We're in business to help clients succeed. Success can mean many things, but, to most businesses, every lost ranking position looks like lost business! As SEOs, SEMs, link builders and inbound marketers, it's important to be successful – not just for our own sake, but also for the companies we work for.

I often tell our clients that it is in our own best interest that they succeed because our success is tied directly to theirs. If they fail, we lose.


But the reverse is also true. If the SEO company fails, the client loses. This is the dual struggle that many companies, including mine, fight against. We want to do everything necessary to make sure our clients succeed. The problem is that not all clients are doing the same for their own success. As the service provider, however, the burden often falls onto us to deliver what is in clients' best interest, even if they are not willing to.

Who's looking out for you/me/we/us?

It's nice to think that doing what's in the client's best interest is really in the client's best interest. Unfortunately, it's not always true. Sad to say, but the “something for nothing” entitlement mentality has taken hold, even among business owners. It's nice that the Internet gives us all kinds of cool things for free. But not all good things are available at no cost. Those that rely solely on what is cheap or free only shortchange themselves.

In SEO, clients may purchase services that provide xyz, based on their budget and hopeful outcome. But when xyz isn't enough to get them where they want, the client then expects the SEO to deliver abc on top of it.

The last thing any SEO wants to hear is, “Why am I not ranking for this keyword?” On the flip side, the last thing the client wants to hear is, “Because you're not paying for that!”

Imagine taking your vehicle to a car wash and paying for the basic wash service. When your car comes out, it's all sparkly clean, except the engine needs to be cleaned, the inside detailed and it could use a coat of wax. Now, the car wash company wants you to have a nice, clean, shiny vehicle. If you're not happy with the wash, you won't be a repeat customer. But it's silly to pay for a wash and then be shocked that the wash alone wasn't enough to make your car look just the way you wanted!

This is what SEOs face all the time. Clients taking a lower service but being upset because they get the lower results!

My best interest is in your best interest

Whenever two companies do business together, their best interests almost universally become one. What is good for you is good for me is good for you.

Businesses come to SEOs because they want (nay, need) to make money. The SEO also wants (nay, needs) to make money. Both need profits to survive. It's in the best interest of both companies when both companies are profitable.

If one is not profitable, it affects the ability of the other to hold up their end of the contract. What would otherwise be a nice long-term and mutually profitable agreement becomes an agreement where, ultimately, neither benefit in the long-run. Your business agreement is out of balance.

If you only look after yourself, then you'll find that no one is willing to work with you, which shortchanges your own efforts. On the other hand, you can't only do what is best for others; otherwise, you won't be able to deliver as promised, which also shortchanges your own efforts.

If you think always doing only what's in your customers' best interest is really the right thing to do, then I challenge you to give away all your services or products free of charge for one year. Wouldn't your clients/customers be better served by such a move? Wouldn't that allow them to get a higher profit margin or keep more money in their pocket?

Of course it would, but it certainly wouldn't be in your best interest. Nor could you do that indefinitely. Sooner or later, your ability to give your clients what they expect will suffer. They'll be unhappy and so will you!

SEO clients need to ensure they are not demanding more than they are willing to pay for, just as SEOs must be willing to do everything they can within the framework provided to get the best results possible. It's in both the SEO's and the client's best interest to ensure they have a mutually beneficial agreement that sets proper expectation for results and that neither takes advantage of the other or allows themselves to be taken advantage of.