One thing that can have a really drastic effect on a large site using Apache, is the value assigned to the MaxClients parameter.

This parameter defines how many simultaneous request can be served. Any connection request from browsers that come in after that will be queued.

Apache prefork, StartServers, MaxSpareServers and MinSpareServers

In the most common case, you will be using Apache in the prefork mode, meaning one process per connection, with a pool of processes pre-forked to standby for connections. The number of spare processes is defined by the values MaxSpareServers, MinSpareServers, while the number to start is defined by StartServers.

Maxclients default

By default, the MaxClients parameter has a compiled in hard limit of 256. This can be changed by recompiling Apache however. Some distributions, or hosting companies raise this limit to a very high value, such as 512 or even 1024 in order to cope with large loads.

While this makes sense when the web server is serving static content (plain HTML, images, ...etc.), it can be detrimental to a dynamic web application like Drupal. So often, we have clients calling because their web server has grind to a halt, and the reason would be a too high MaxClients value.

A web site's nemesis: Excessive Thrashing

The reason is that if your web site experiences a traffic spike, or if there is a bottleneck in the database, incoming requests cause new processes to be forked at a rate higher than old processes can service the older connections. This causes a condition where the system keeps creating new processes that overflow the available memory and starts to use the swap space. This almost always causes thrashing, where the system is just swapping pages from physical memory to virtual memory (on disk), and vice versa, without doing any real work. You can detect if thrashing has occurred by using the vmstat command (see our page on tools for performance tunings and optimization for more info).

A simple calculation for MaxClients on a system that does only Drupal would be:

(Total Memory - Operating System Memory - MySQL memory) / Size Per Apache process.

If your hosting company configured your server with all sorts of bells and whistles (like mod_perl, mod_python, in addition to mod_php), then Apache can easily be 21 MB per process. If your server has 512MB, then you can fit some 20 Apache processes. If you tune Apache well, and remove all the unneeded modules, and install a PHP op-code cache/accelerator, then you can make each Apache process take as little as 12 MB. These figures depend on how many modules you have loaded, how big they are, so there is no hard and fast rule. Even if one has 1GB of memory, and leaves 250 MB for the system and MySQL, with an Apache process of 15MB, this means 50 Apache processes can fit in the remaining 750MB.

Remember that you need memory for the operating system, as well as for MySQL. The more you give the system and MySQL memory, the more caching of the file system they do for you and avoid hitting disk, so do not use the very last available memory for MaxClients.

Tuning the ServerLimit 

On some systems, there is another parameter that  sets an upper limit if MySQL. So for example, if ServerLimit is set by default to 256, and you want to increase MaxClients to 300, you will not be able to do so, until you set ServerLimit to 300 as well. Normally, you would see a warning message from Apache when you restart it to tell you that this needs to be done.

Conclusion

If you cannot do a proper calculation, then it is safest to start with a conservative number, e.g. 60 to 150 on a 2GB system, and then increase it as you monitor the usage of the system over a few weeks. By all means, do not keep it at the 512 value that came with your server/distribution until you know how much load you can handle.

Resources and Links

Articles

Comments

Visitor (not verified)

Tue, 2008/10/21 - 19:28

Quote:

If your hosting company configured your server with all sorts of bells and whistles (like mod_perl, mod_python, in addition to mod_php), then Apache can easily be 21 MB per process.
== end of quote==

How do I know what apache modules are required to run drupal websites?
we have a dedicated server and root access, and we would like to have the apache modules configured specifically for drupal.

is there a "list" of necessary apache modules that we should have, while disabling all others?

Thanks for such a lovely article

Visitor (not verified)

Tue, 2009/02/03 - 21:46

when making this calculation, how do i accurately find my size in MB of each apache process? for example, if i run top -d 1 and look at the results, i have different sizes for each process: VIRT, RES, and SHR. it looks like in top, my total memory usage is in line with SHR, the smallest of the memory numbers. is it safe to make this calculation based on apache process SHR size?

Visitor (not verified)

Thu, 2009/06/18 - 10:20

Thanks man you really solved my problem
i am always do wrong
for maxclient
and serverLimit
thanks for share

Visitor (not verified)

Fri, 2009/09/18 - 02:20

No matter how we try, the process size is always around 60M:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                
20808 apache    15   0  362m  58m  21m R 27.4  0.7   0:03.20 httpd                              
20823 apache    16   0  402m  86m  23m S 10.7  1.1   0:01.67 httpd                          
20811 apache    16   0  348m  43m  21m S  3.8  0.5   0:02.48 httpd                          
20826 apache    16   0  362m  52m  21m S  3.4  0.7   0:02.34 httpd

I guess this is also influenced by php memory limit we have? (96M)

foxtrotcharlie (not verified)

Mon, 2011/07/11 - 01:48

Our site uses Apache's prefork MPM. I have set MaxClients to 75 but when doing some ab tests the number of processes reported in Munin reaches 150.

My question is: are the 75 processes over my MaxClients limit actually being used (and therefore MaxClients is not being respected), or are these additional processes just queued and won't have any impact on RAM usage?

Thanks for the very useful article :-)

emil (not verified)

Wed, 2012/08/08 - 16:40

I'm using godaddy dedicted server by far the worst service and knowledge they destroit all my drupal websites and provide no solutions after charging me over $1000 in support, they can't even fix my sql or to create new password and user name to SSH (shell access and etc..) they said we try it all but they will not refound my money for selling me bad product and bad service, i'm stuck i need neww server that can handel high cpu and can handel large database with no to much effort needed, i need solution for large drupal website with over 5 milions pages and that can handel high traffic 24 hour 365 days a year with out all those crushes and without to many optemization needs for chace and cpu any idea?

my server giving me this error and i can't even change the max_allowed_packet since i have no way to access the shell for my server and godaddy can't restet the password to the own products that they are selling please help and please give any idea where to go to buy this kind of server that i need base on the information above.

Additional uncaught exception thrown while handling exception.
Original
PDOException: SQLSTATE[HY000]: General error: 1030 Got error -1 from storage engine: DELETE FROM {semaphore} WHERE (name = :db_condition_placeholder_0) AND (value = :db_condition_placeholder_1) AND (expire <= :db_condition_placeholder_2) ; Array ( [:db_condition_placeholder_0] => menu_rebuild [:db_condition_placeholder_1] => 170783664650214bef7b50e4.73991544 [:db_condition_placeholder_2] => 1344359448.36 ) in lock_may_be_available() (line 181 of /home/clean123/public_html/includes/lock.inc).

Additional
PDOException: SQLSTATE[HY000]: General error: 1467 Failed to read auto-increment value from storage engine: INSERT INTO {watchdog} (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9); Array ( [:db_insert_placeholder_0] => 1 [:db_insert_placeholder_1] => php [:db_insert_placeholder_2] => %type: !message in %function (line %line of %file). [:db_insert_placeholder_3] => a:6:{s:5:"%type";s:12:"PDOException";s:8:"!message";s:422:"SQLSTATE[HY000]: General error: 1030 Got error -1 from storage engine: DELETE FROM {semaphore} WHERE (name = :db_condition_placeholder_0) AND (value = :db_condition_placeholder_1) AND (expire &lt;= :db_condition_placeholder_2) ; Array ( [:db_condition_placeholder_0] =&gt; menu_rebuild [:db_condition_placeholder_1] =&gt; 170783664650214bef7b50e4.73991544 [:db_condition_placeholder_2] =&gt; 1344359448.36 ) ";s:9:"%function";s:23:"lock_may_be_available()";s:5:"%file";s:44:"/home/clean123/public_html/includes/lock.inc";s:5:"%line";i:181;s:14:"severity_level";i:3;} [:db_insert_placeholder_4] => 3 [:db_insert_placeholder_5] => [:db_insert_placeholder_6] => http://123carpetclean.com/ [:db_insert_placeholder_7] => [:db_insert_placeholder_8] => 76.93.5.103 [:db_insert_placeholder_9] => 1344458134 ) in dblog_watchdog() (line 154 of /home/clean123/public_html/modules/dblog/dblog.module).

Jean-Sebastien ... (not verified)

Tue, 2012/11/20 - 21:16

A few weeks ago I needed to tune an apache web server for a client, and although the process is straight-forward (see memory used by processes, server memory available, etc.), it's also a bit tedious.

I expected to find a ready made application / script to do the number crunching, but my search didn't turn up anything, so I ended up writing my own. ;-)

I figured you (and your readers) might find it useful as well.

You can find the weblog entry here and the script on Google Code as well.

Enjoy!

js.

Is your Drupal or Backdrop CMS site slow?
Is it suffering from server resources shortages?
Is it experiencing outages?
Contact us for Drupal or Backdrop CMS Performance Optimization and Tuning Consulting