Google Crawler hitting your site too aggressively?
If your Drupal site suffers occasional slow downs or outages, check if crawlers are hitting your site too hard.
We've seen several clients complain, and upon investigation we found that the culprit is Google's own crawler.
The tell tale sign is that you will see lots of queries executing with the LIMIT clause having high numbers. Depending on your site's specifics, these queries would be slow queries too.
This means that there are crawlers that accessing very old content (hundreds of pages back).
Here is an example from a recent client:
Most Comments