Googlebot can’t access your site – Don’t panic

googlebot errors to ignore

Googlebot feeds search content to Google (Image Michael Cote via Flickr)

Every now and then, a client emails me to say they’ve received a message from the ‘Googlebot’ claiming it cannot access their website.

The first thing to say is, don’t panic.

If you get a very occasional message like this, once or twice a year, all is fine.

What it typically means, when I have followed up in Google Webmaster Tools to investigate is that on a particular occasion the Google spiders didn’t report back to Google that they’d had a successful visit to your site.

There can be all manner of reasons for this.

The short answer is, wait to see if you get another message.

If you get two or three of these messages on CONSECUTIVE days, then there is an issue worth investigating.

What blocks the Googlebot?

There are a number of things that might cause the Googlebot from being delayed or blocked.

One of them is just the vagaries of the internet itself; traffic congestion and intermittent server issues at your end or at Google’s end (yes, it does happen).

Meanwhile, there are things webmasters sometimes do to deliberately block Googlebot during website builds and then forget to lift, namely, a setting in WordPress called “Discourage search engines from indexing this site” found in Settings > Reading. Make sure that is not ticked. If it is, it will be politely asking search engines like Googlebot to ignore your site.

The same ‘not welcome’ sign can be found on your server as a robots.txt file which asks search engines to ignore your site. Type in your domain/robots.txt to see if anything appears. Hopefully it won’t. If it does, ask your webmaster to delete it.

Despite all those options, typically, when you get the Googlebot warning message, just wait to see if it appears again. If it does, then it is time to log into your Google Webmaster Tools account and look for Crawl Errors where Google will explain all (if your WordPress site was built by Baker Marketing you will automatically have a Google Webmaster Tools account).

But, again, only bother with that after some consecutive messages. Most likely it was a one off and your time would be better spent crafting fresh marketing material aka a new blog post!

Previous

Next

3 Comments

  1. Simon Elliot

    Hi Steve.

    Our website features multiple language versions, organised in a sub folder structure.

    domain.com/fr
    domain.com/es
    domain.com/ch

    Our robots.txt is on the root, however via WMTs I keep seeing errors where Googlebot has tried to access the robots.txt file on the sub folder… i.e domain.com/fr/robotx.txt

    Now Google only read the robots.txt on the root.

    So why are they trying to access URLs including:

    domain.com/fr/robots.txt
    domain.com/es/robots.txt
    domain.com/ch/robots.txt

    And why are they saying “We couldn’t access your robots.txt file” ?? They aren’t looking in the right place!

    I’ve already inserted the robots.txt file into the main root, and it has been working for the last 2 years. This appears to be an issue with Webmaster tools THINKING that domain/fr (which is geotargeted to France) is a root, and should therefore have a robots.txt file.

    The WMT’s error I’ve received say that “googlebot can’t find a robots.txt file at domain.com/fr/robots.txt” which is very strange.

    My options are therefore:

    a) 301 redirect all requests for domain.com/fr/robots.txt to domain.com/robots.txt
    b) Ignore the issue, and let googlebot solve it.

    Any ideas here?

    Reply
    • Steve Davis

      Hi Simon

      I have to say that I am a marketer first who is part Geek, rather than a full geek.

      In short, I think your last option is sound IF your various language sub-sites are still performing for you as you would expect.

      If they have taken a hit and not performing, then it will be worth connecting with full Geek to ‘help’ the Googlebot, in my opinion.

      I hope that helps.

      Steve

      Reply
  2. milad

    very goooooood

    Reply

Trackbacks/Pingbacks

  1. How a WordPress pingback helped me catch a plagiarist - [...] A very popular post in this WordPress Wednesday series is the article, Googlebot can’t access your site – Don’t panic.…

Submit a Comment

Your email address will not be published. Required fields are marked *

Chat With Us On Messenger