Recently a client complained that their site has been suffering from slow downs that were not there before.
Upon investigating the history of resource utilization, we found that memory usage has increased, and at the same time, the load average has increased as well.
To cut a long story short, we found out that initially, the site had neither dblog nor syslog modules were enabled, and they were enabled before the issues started. They started when syslog was enabled.
Normally, this would not be an issue on a normal site. But for this site, there were lots of PHP and module errors that were written to the log, that it did tie the disk.
These error are like so:
purl: Infinite redirect prevented.
Warning: Invalid argument supplied for foreach() in some_module_category_context() (line 539 of .../modules/custom/some_module/some_module.module).
We pointed out these errors to the client, so that they fix their code.
For the meantime, and to test/prove that this is the root cause, we disabled the syslog module.
Upon doing so, memory utilization went down from 13 GB to 4GB. And the load average dropped from 4 to 5, down to a mere 1 to 2.
The graphs show the decrease in CPU, load average and specially disk I/O and memory usage.
CPU graph. Note the height of the green peaks over the week.
Load average graph.
Disk I/O is much less now.
And memory is also less. The daily cycle is caused by the log rotation of the Drupal syslog. Now the usage is much less, and therefore more free memory for file system cache and buffers.
So, if you have a high load average and the site is slow, check if you are writing a lot of data to the disk either via dblog or syslog. That may be your issue.
Comments
Droplet (not verified)
I did some test before and
Sun, 2011/11/27 - 18:24I did some test before and seems like drupal error handler also slow it down: http://drupal.org/node/1138464
dalin (not verified)
Moral of the story: syslog
Mon, 2011/11/28 - 13:19Moral of the story: syslog module is only useful if you actually stay on top of it and deal with the errors that it is logging.
Khalid
Yes
Mon, 2011/11/28 - 13:46Yes, that is it in a nutshell.
Even if you are using dblog, and have lots of errors, this will overload the database because of a lot of writes to the disk.
Using syslog in that case just shifts the problem from one layer to another, and would still be writing to some disk in the end lots of crap for each page load.
The unexpected part is the increased memory usage when this happens.
I added graphs to show the difference.
Visitor (not verified)
No. Just NO!
Thu, 2011/12/29 - 05:43Moral of the story: don't have php-errors and notices in the first place.
If your stakes are high enough to be troubled by a large amount of PHP notices and errors, you should seriously consider moving to software that does not have that many errors and notices in the first place. The fact that these notices and errors are there -Actually : the fact that there are so many there that writing them to disc causes your servers to slow down- should be enough reason to run away from that software screaming and shouting.
Surpressing notices is utter crap. They should not be there in the first place.
Moral of the story: use properly written software.
Khalid
Always the perfectionist ...
Thu, 2011/12/29 - 11:46Ber, always the perfectionist ...
Life is more complicated than that though. Sometimes, the choices do not include fixing the software for various reasons.