The most dreadful thing happened recently, something that any website or blog owner hopes would never happen. My blog literally dropped out of Google. Zilch! Nada! Nichts! 無 ! Rien!…you get the picture right.

Well not anymore, I’m happily back into Google and they love me as much as they did before. But for a few excruciating days my search results, for the terms I usually get traffic for, were dropping one by one like flies hit with a can of bug spray.

So what happened? The same thing apparently happened to John Chow where he probably attributed it to a Google algorithm update, only to later return back as usual in the SERP’s. Some SEO Guru also apparently tried to swindle $4,000 from him without a guarantee of results, some people these days - srsly!

For illustration purposes here is a graph depicting the demise from May 28th onwards, to the rebound on the 5th of June. {I don’t like to give my traffic numbers away, unless it’s for advertising.}

So John resisted from saying how he happened to fix his rankings on his blog post, reasoning that it is probably too valuable an information to reveal. Maybe I had the same problem he had and probably applied the same fix, I’m not saying this is how he did it. I’m saying this is how I did it.

But first…

What search terms was I was highly ranking for? Well a lot of long tail key phrases and quite a few two - three word search terms. Like video hybrid, online videos, lacoste sale and many more. So when I noticed a sudden drop to my most visited pages I got thrown off. Doing a search for the number of pages that were indexed for my blog in Google showed no variation, so what went wrong? Obviously I was not banned

Google Webmaster Tools - A friend in need is a friend indeed

I logged into my Webmaster tools account and saw that from the “0″ URL’s that were restricted by robots.txt it had now jumped to 800+. YIKES!

The robots.txt file for this blog was adjusted around the 28th of May, in the attempt at eradicating duplicate content and trying to get out of supplemental results, I had inadvertently just dumped my entire site, except for the home page.

This is what I had added to my robots.txt file

Disallow: /2006/0*
Disallow: /2006/1*
Disallow: /2007/0*
Disallow: /2007/1*

Do NOT do this to your robots.txt file especially if your wp permalinks are set up with year, month and day as this blog is.

What a goof!

That’s probably what you are thinking right now. Which is fine, I accept that I goofed. Live and learn I say. I will say that I foolishly copied the “Disallow:” lines from another site.

So now my robots.txt only disallows the feed/, trackback/ along with the category/ and page/ as most of this is duplicate content and not actual individual blog posts. Also I’ve disallowed the Googlebot-Image as I don’t like people scrapping for images, make your own! I also name them weirdly so they have no relevancy to any typed in image search term from other engines.

Bouncing Back

I implemented the change on the 3rd of June and slowly, but surely, my blog is coming back into the search result pages and my traffic is going back up again.

Logging into Google webmaster tools I see that since June 4th only the /feed/, /trackback/ and /page/ pages of the blog are not indexed. Just as I wanted.

So if this was the same reason, with the robots.txt file, that John Chow’s blog dropped out of the search results then that is how it happened, and probably that’s how he fixed it too. If not, then he did something else of which I have no knowledge.

Is it valuable information?

Well if someone tried to swindle $4,000 from him because of that then it probably is. But to me it isn’t, Why? Because it isn’t really something that anybody who does a little investigating can’t figure out by themselves.

Google themselves provide you all the tools necessary to make sure that your site is indexed properly, so take advantage of them.

Sign up for my full RSS feed.

About the author

Hyder has been blogging for the past two years on this blog. He started Weborithm, his web company, in 2007 and along with designing blogs and websites also releases various web related products.