Identifying aggressive crawlers using Go Access
Aggressive crawlers that hit your web site a lot can cause performance problems.
There are many ways to identify aggressive crawlers, including writing custom scripts that analyze your web server logs.
One tool that we found to be useful in analyzing which crawlers hit the site the most today or yesterday is Go Access.
Getting Go Access
Go Access is available for Ubuntu Natty Narwahl (11.04) only, but not earlier LTS releases.
Most Comments