googlebot errors to ignore

Googlebot feeds search content to Google (Image Michael Cote via Flickr)

Every now and then, a client emails me to say they’ve received a message from the ‘Googlebot’ claiming it cannot access their website.

The first thing to say is, don’t panic.

If you get a very occasional message like this, once or twice a year, all is fine.

What it typically means, when I have followed up in Google Webmaster Tools to investigate is that on a particular occasion the Google spiders didn’t report back to Google that they’d had a successful visit to your site.

There can be all manner of reasons for this.

The short answer is, wait to see if you get another message.

If you get two or three of these messages on CONSECUTIVE days, then there is an issue worth investigating.

What blocks the Googlebot?

There are a number of things that might cause the Googlebot from being delayed or blocked.

One of them is just the vagaries of the internet itself; traffic congestion and intermittent server issues at your end or at Google’s end (yes, it does happen).

Meanwhile, there are things webmasters sometimes do to deliberately block Googlebot during website builds and then forget to lift, namely, a setting in WordPress called “Discourage search engines from indexing this site” found in Settings > Reading. Make sure that is not ticked. If it is, it will be politely asking search engines like Googlebot to ignore your site.

The same ‘not welcome’ sign can be found on your server as a robots.txt file which asks search engines to ignore your site. Type in your domain/robots.txt to see if anything appears. Hopefully it won’t. If it does, ask your webmaster to delete it.

Despite all those options, typically, when you get the Googlebot warning message, just wait to see if it appears again. If it does, then it is time to log into your Google Webmaster Tools account and look for Crawl Errors where Google will explain all (if your WordPress site was built by Baker Marketing you will automatically have a Google Webmaster Tools account).

But, again, only bother with that after some consecutive messages. Most likely it was a one off and your time would be better spent crafting fresh marketing material aka a new blog post!

Pin It on Pinterest

Share This