Monday, November 20, 2006

Googlebot Gone Wild

I've narrowed down my server problems to a ridiculous amount of traffic coming from the Googlebot... I want Google to visit and index lots of pages of sketchy.com, but I had to request that they chill the Googlebot out. I've also revised the sketchy.com robots.txt file to try and slow msnbot, Slurp, Googlebot and Speedy down, as well as stop them from crawling dynamic pages. I've blocked a bunch of shady bots in my .htaccess as well.

0 Comments:

Post a Comment

Links to this post:

Create a Link

<< Home