Apache with fcgid: acceptable performance and better resource utilization

Published Tue, 2009/02/24 - 19:27, Updated Tue, 2014/07/22 - 02:29

Update May 2013: We no longer recommend fcgid ever since Ubuntu Server 12.04 was released. This is because that version has PHP-FPM, which provides every benefit that fcgid has, with the added advantage of a shared opcode cache for all processes. We will be writing a full article on PHP-FPM with Drupal in the near future (soon to appear at High Performance Drupal with Apache MPM Worker Threaded Server and PHP-FPM).

Most sites that serve large Drupal instances use Apache with mod_php. Although this is often the easiest and fastest option, it may not be the ideal situation in some cases. This article explains fcgid, a way to run PHP other than mod_php, how to get it configured, and what are the advantages and disadvantages of it.

Apache's mod_php

Apache's mod_php is the most widely used mode for PHP with Apache. mod_php itself is the entire PHP interpreter embedded in each Apache process that gets spawned. This provides performance and stability benefits, e.g.

  • No need to call an external process (e.g. CGI).
  • No need to communicate with another process via sockets (e.g. Fast CGI).
  • The APC cache is shared by all Apache processes.

It also has some disadvantages

  • The memory footprint per Apache process is large, specially when sites indulge in contributed modules.
  • If Apache is serving static content, e.g. images and CSS files, it still has to spawn large processes because of the embedded PHP interpreter.

CGI

CGI (Common Gateway Interface) is the legacy way of runing applications on the web from the mid 1990s or so. It was too inefficient for anything but small sites. CGI spawns a new process for every incoming request to execute a PHP script, a very resource intensive and inefficient way of doing things. No wonder it faded away over time as web applications became more complex.

FastCGI

FastCGI was introduced to avoid some of the issues with running languages, including PHP, inside the Apache process, as well as avoiding the inefficiency of CGI.

A FastCGI application is executed outside of the web server (Apache or other wise), and waits for requests from the web server using a socket. The web server and the FastCGI application can even be on separate physical machines and communicate over the network.

Because the web server adn the application processes are separate better isolation is possible.

In reality, running PHP as mod_fastcgi with Apache has proved to be problematic. Mainly with stability. Even on Drupal.org we tried it for a while, but switched back to mod_php after some time.

mod_fcgid

mod_fcgid was introduced to be binary compatible with FastCGI, but with better control over spawning processes. The benefits of process isolation are still there.

Installing fcgid

The article assumes you are using Ubuntu/Debian. For Red Hat or Centos, use the equivalent packages and yum to achieve the same results.

First we install the required Apache components, with Apache threaded server (MPM Worker), to save memory.

aptitude install apache2-mpm-worker libapache2-mod-fcgid

Then we enable the Apache fcgid module.

a2enmod fcgid 

And install PHP CGI, and the a few other PHP components, if they are not already on your system.

aptitude install php5-cgi php5-curl php5-gd php5-mysql

Configuring fcgid

To configure fcgid, you have to do two things:

1. Create a new file in /etc/apache2/conf.d/php-fcgid.conf and put the following in it:


  AddHandler fcgid-script .fcgi .php
  # Where to look for the php.ini file?
  DefaultInitEnv PHPRC        "/etc/php5/cgi"
  # Maximum requests a process handles before it is terminated
  MaxRequestsPerProcess       1000
  # Maximum number of PHP processes
  MaxProcessCount             10
  # Number of seconds of idle time before a process is terminated
  IPCCommTimeout              240
  IdleTimeout                 240
  #Or use this if you use the file above
  FCGIWrapper /usr/bin/php-cgi .php



  ServerLimit           500
  StartServers            3
  MinSpareThreads         3
  MaxSpareThreads        10
  ThreadsPerChild        10
  MaxClients            300
  MaxRequestsPerChild  1000

For a large site with a server with more memory and CPUs we can use this:


  AddHandler fcgid-script .fcgi .php
  # Where to look for the php.ini file?
  DefaultInitEnv PHPRC  "/etc/php5/cgi"
  # Where is the PHP executable
  FCGIWrapper /usr/bin/php-cgi .php
  # Maximum requests a process handles before it is terminated
  MaxRequestsPerProcess 1500
  # Maximum number of PHP processes.
  MaxProcessCount       45
  # Number of seconds of idle time before a process is terminated
  IPCCommTimeout        240
  IdleTimeout           240


# Large site

  ServerLimit          2048
  ThreadLimit           100
  StartServers           10
  MinSpareThreads        30
  MaxSpareThreads       100
  ThreadsPerChild        64
  MaxClients           2048
  MaxRequestsPerChild  5000

2. Add ExecCGI to the Options line in the vhosts you want to use PHP and or fcgid on, for example. 

Order Allow,Deny
Allow From All
Allow Override All
Options MultiViews Indexes Includes FollowSymLinks ExecCGI

Now you should restart Apache, and fcgid should be active.

Memory consumption

The first thing you observe is that the memory size of the fcgi process is considerably larger (40MB) than when PHP is running as a mod_php within Apache (31MB). However, when using fcgi, Apache (pre-fork) is only 3.3 to 2.4 MB per process so you can have more of them to serve static files. The savings are even more if you use the threaded server. For example, an Apache process on a large site would be 20 to 25 MB, and the total number needed for thousands of threads would be 10 to 15 processes total. You also normally have less CGI processes in total than Apache.


For example:

Apache pre-fork
fcgi 15 processes * (40 MB - 21 MB shared) + 21 MB shared + (20 Apache procs * 6 MB) = 426 MB
mod_php 26 processes * (26 MB - 13 MB shared) + 13 MB shared = 351 MB

Apache threaded MPM Worker
fcgi 15 processes * (40 MB - 21 MB shared) + 21 MB shared + (10 Apache procs * 25 MB) = 290 MB
mod_php 26 processes * (26 MB - 13 MB shared) + 13 MB shared = 351 MB

The exact figures will vary for your site, depending on how you configured Apache and what modules you have enabled on your Drupal site.

Maximum number of processes

Setting the upper limit for the number php-cgi processes is possible but tricky. Normally MaxProcessCount should be the parameter to set to prevent creation of PHP processes above the number set for that parameter. I had to dig in the mod_fcgid source to find that parameter.

Make sure that you set this figure to a reasonable one. Setting it too low will cause an error to be logged in your web servers error log, for example:

[Sun Jul 25 17:06:23 2010] [notice] mod_fcgid: /home/example.com/www/index.php total process count 25 >= 25, skip the spawn request

On a 512MB VPS for example, a value of 7 to 10 seems to be adequate. For an 8GB dedicated server, a value of 45 or even higher will do. The exact figures will depend on how many modules you have, and how memory they consume.

Also, make sure that you do not set PHP_FCGI_CHILDREN at all. What happens is that fcgid uses it as a multiplier, not an upper maximum, and PHP will kept on breeding like rabbits, and the server will thrash and swap very quickly.

Performance benchmarking

We conducted performance benchmarks comparing mod_php and mod_fcgid. The conclusion is that mod_php is a bit faster than fcgid, but not by much.

Our benchmarks how the following at different concurrency levels:

With 4 concurrent users
fcgid
Transactions:                    826 hits
Response time:                  0.58 secs
Transaction rate:               6.90 trans/sec
Successful transactions:         826
Longest transaction:            2.13
Shortest transaction:           0.26
Apache mod_php
Transactions:                    853 hits
Response time:                  0.56 secs
Transaction rate:               7.12 trans/sec
Successful transactions:         853
Longest transaction:            0.95
Shortest transaction:           0.26
With 15 concurrent users:
fcgid
Transactions:                    674 hits
Elapsed time:                 120.35 secs
Response time:                  2.65 secs
Transaction rate:               5.60 trans/sec
Concurrency:                   14.83
Successful transactions:         674
Longest transaction:            7.28
Shortest transaction:           2.23
Apache mod_php
Transactions:                    748 hits
Elapsed time:                 120.42 secs
Response time:                  2.40 secs
Transaction rate:               6.21 trans/sec
Concurrency:                   14.90
Successful transactions:         748
Longest transaction:            4.19
Shortest transaction:           1.80 

Hunter Scott Newman  confirmed the benchmarks independently, and sent us the following results. Note that he is using the threded Apache server with fcgid.

Apache Worker + fcgid
4 users
Requests per second:    391.59 [#/sec] (mean)
Time per request:       10.215 [ms] (mean)
Time per request:       2.554 [ms] (mean, across all concurrent requests)
10 users
Requests per second:    585.91 [#/sec] (mean)
Time per request:       17.067 [ms] (mean)
Time per request:       1.707 [ms] (mean, across all concurrent requests)
50 users
Requests per second:    647.78 [#/sec] (mean)
Time per request:       77.187 [ms] (mean)
Time per request:       1.544 [ms] (mean, across all concurrent requests)
Apache Prefork + mod_php
4 users
Requests per second:    445.88 [#/sec] (mean)
Time per request:       8.971 [ms] (mean)
Time per request:       2.243 [ms] (mean, across all concurrent requests)
10 users
Requests per second:    655.13 [#/sec] (mean)
Time per request:       15.264 [ms] (mean)
Time per request:       1.526 [ms] (mean, across all concurrent requests)
50 users
Requests per second:    670.00 [#/sec] (mean)
Time per request:       74.627 [ms] (mean)
Time per request:       1.493 [ms] (mean, across all concurrent requests) 

Benefits and advantages

There are several advantages for using fcgid. Performance is not one of them, but the penalty may be well worth it for the following advantages:

  • Less memory consumption. Apache processes that serve static files are very small (3MB or less). So we can have more of those in memory.
  • Less network connections active, since there are less PHP process
  • Less database connections active, since there are less PHP processes. This is a very important point, since MySQL performance drops as the number of connections increase.

Case study: Large site with fcgid

In order to show what fcgid can help with, consider the following graphs. See the before and after comparison.

Memory usage by day, before and after.

Memory usage fcgid - day

The same was true over a week too.

Memory usage fcgid - by week

And the number of MySQL threads went down dramatically, because only PHP processes will connect to it, and not every Apache process.

fcgid MySQL threads

And also the number of network connections inside the server.

fcgid - Network connections

And just to make our lives easier, we wrote a monitoring script that will plot the number of php-cgi processes on a graph. We can know what is going on and what was going on too.

Munin php-cgi monitor

Caveats and precautions

Not everything is rosy though. There has to be some drawback.

APC still stores the op-code caches for Drupal and its modules. It has to do the parsing more often though, unlike on mod_php.

Because of that, you may on occasions see errors such as "Fatal error: Allowed memory size of nnnnnn bytes exhausted (tried to allocate nnnn bytes)". Reloading the page causes it to load normally.

Although we have not dug too deep into it, we have found is that APC is not shared among the processes. And because the php-cgi processes are killed and new ones spawned, the persistence of APC's user/data cache cannot be relied upon. This is not important for most users, but if you are using things like the Drupal cache router module or the performance logging module, this will impact you.

Conclusion

If pure speed is what you are after, then stay with mod_php.

However, for better resource usage and efficiency, consider moving to fcgid.

Resources

Here are some other links on fcgid. Note that some of the configurations mentioned recommend the use the PHP_FCGI_CHILDREN and other settings that proved disastrous. They also do not mention the MaxProcessCount parameter which we found from the source code.

Update: Article updated to use Apache threaded server (MPM Worker) instead of the pre-fork server. This provides lots of memory savings.

How can this work with plesk

How can this work with plesk installed on the vps container?

I play at this one

I think the advantages of

I think the advantages of Apache's mod_php explains it's remarkable features. But the disadvantages significantly points out it's limitations.

visit site

Apache fcgid module already enable

Hello,

this is a great explained article but i have a question, when i try to enable fcgid it says it have already been enabled so that mean i already have that module, so should i continue with the rest of the process? also i looked in to vhost.conf and i can see

Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch +ExecCGI
allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch
AddHandler fcgid-script .php
AddHandler fcgid-script .php5
FCGIWrapper /home/seekdl/fcgi-bin/php5.fcgi .php
FCGIWrapper /home/seekdl/fcgi-bin/php5.fcgi .php5

allow from all
AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,Indexes,MultiViews,SymLinksIfOwnerMatch

can i still continue with the step 01) 1. Create a new file in /etc/apache2/conf.d/php-fcgid.conf and put the following in it:

Kindly let me know

Thank you

What timeout?

For your apache configuration, do you use Keealive?

What are the values you use for keepalivetimeout, and for apache timeout?

Thank you in advance.

No longer recommend fcgid

We no longer recommend fcgid. Ever since Ubuntu 12.04 was released, we recommend PHP FPM, which is in the repositories. We will be writing an article on it in the near future.

For timeouts, the following works well for most sites:

KeepAlive On
MaxKeepAliveRequests 30
KeepAliveTimeout 3
Timeout 5

But, isn't php-fpm bad if

But, isn't php-fpm bad if i've for example a vps with 10 sites from different clients?

My apache worker process size is around

i using MPM worker, i found better performance with worker
but when i checked the mem usage, i found that 1 apache process is using 200MB

Why this,

my worker module conf

StartServers 1
MaxClients 40
MinSpareThreads 10
MaxSpareThreads 40
ThreadsPerChild 20
MaxRequestsPerChild 1000

i installed apc also

check this link http://discusswire.com/problem.png

I am not finding this to be a great setup with multiple sites..

Hi,

Thanks for the article...

I have switched from php-cgi to using mod_fcgid to get better performance but its using far more memory because I am not running a single site on the server..

Each site is spawning its own php processes and these processes aren't shared between sites even though they are all running running as the "www-data" user.. So if I go to site1.com a new php process is started and uses about 30MB of memory.. Then I go to site2.com and another 30MB of RAM allocated to a second process.. So by the time I have visited 5 sites on the server I have chewed up ~150MB of memory in php processes..

I have set FcgidMinProcessesPerClass 1 so additional processes per site are killed off but I am still sitting with 1 running per site rather than the php processes being shared between sites and spawning and killing additional as needed.. Setting FcgidMinProcessesPerClass 0 seems pointless because it would effectively then be similar to running php-cgi when php processes are spawned and killed as needed anyway and still wouldn't solve the high memory usage issue when all sites are being accessed because its not "pooling" the php processes..

If you have any thoughts on how this can be resolved so that running php processes are shared between sites please let me know..

Yes, and No ...

This setup is used on mainly dedicated servers (4GB or more), or VPS's with sites with a low number of modules (we do use it on 512MB VPS's with no issues.

We do use Ubuntu Server, and the setup we described works well without specifying per class and all that.

I don't think that a PHP process cares which site it is serving per se. But perhaps not spawning a new process is depending on keep alive?

For fcgid (or any FastCGI), the main issue is that the APC code cache is not shared across processes. So it has to replicate this cache for each process, unlike mod_php, which shares the cache. The size of each process is dependent on the number of modules that are used, whether they are true multisite (i.e. sites/all/modules is the same for all sites or not), and the APC cache.

Using less modules will reduce the memory footprint of each process.

You can find out how much APC is using if you use the apc.php script and tune it accordingly.

You need to find the sweet spot where FcgidMaxProcesses is set to a value that does not overflow memory, yet have enough processes to serve the site efficiently. For large sites on dedicated machines, we use 35 or 40. Less for smaller servers with constrained memory.

Hi, It seems that the server

Hi,

It seems that the server (hardware configuration) itself isn't specifically relevant.. Its more to do with the number of sites running on the server.. It would appear that each site (apache virtual host) is its own "class" as defined by fcgid and so a new fcgid/php process is spawned for each site..

I have tested your theory about keepalive holding on and can confirm that even waiting a long time and closing the browser doesn't allow the different sites to use the same php process.. It still creates a new one if one isn't already running for that site..

As regards APC, its simply not an option for smaller servers that aren't running mod_php.. I am investigating xcache which it seems might share memory across processes spawned for the same site.. I doubt there will be any sharing between processes spawned for different sites..

A lot of this may be the reason that the majority of articles about Drupal optimisation are centred around mod_php as the php handler so that APC is viable and with the claimed performance gains the apache resource utilisation and blocking factors when running mod_php are negated..

512MB VPS with 4 sites

We've run this configuration on 512MB VPS with 4 sites on it, and never saw that issue you are facing.

Again, we don't use any class parameters.

Here is the configuration used, on Ubuntu Server 8.04 LTS:

  AddHandler fcgid-script .fcgi .php
  DefaultInitEnv PHPRC        "/etc/php5/cgi"
  MaxRequestsPerProcess       1000
  MaxProcessCount             5
  IPCCommTimeout              600
  IdleTimeout                 600
  FCGIWrapper /usr/bin/php-cgi .php

We run a different setup for several high traffic sites that get up to 3 million page views a day, with no issues. But these large sites are one site per server anyway due to the traffic levels.

Hi, I changed my config to be

Hi,
I changed my config to be exactly the same as what you have just posted but I am definitely still seeing a separate fcgid/php process per apache virtual host..

Also strangely with your config but with FcgidMaxProcesses 3, I was able to visit 5 sites and start 5 fcgid processes which seems to suggest the FcgidMaxProcesses is being ignored somewhere somehow..

I will setup a fresh Ubuntu 10.04 server tomorrow and configure it from scratch and see if I can repeat this behaviour on a new server..

Differences

We found differences between 10.04 and 8.04.

Here is the configuration on Ubuntu Server LTS 8.04, Xen VPS, 512MB, 4 sites.

  AddHandler fcgid-script .fcgi .php
  DefaultInitEnv PHPRC        "/etc/php5/cgi"
  FCGIWrapper /usr/bin/php-cgi .php
  MaxRequestsPerProcess       1000
  MaxProcessCount             5
  IPCCommTimeout              600
  IdleTimeout                 600

Here is the configuration on Ubuntu Server LTS 10.04, dedicated, 16GB, single site.

  AddHandler fcgid-script .fcgi .php
  FcgidInitialEnv PHPRC  "/etc/php5/cgi"
  FcgidWrapper /usr/bin/php-cgi .php
  FcgidMaxRequestsPerProcess 5000
  FcgidMaxProcesses            40
  FcgidIdleTimeout            300
  FcgidIOTimeout              300
  FcgidProcessLifeTime        600

You can see that we found differences on which parameters restrict the maximum number of processes so we don't overflow memories.

Note that one is a single site, and the other is multi-site though.

Thanks for the examples.. I

Thanks for the examples..

I have confirmed that each virtual host does in fact start a new fcgi process.. Also by reading the mod_fcgid page on apache.org it details the following..

Certain settings or other concepts that depend on the virtual host, such as FcgidInitialEnv or process classes, distinguish between virtual hosts only if they have distinct server names. (See the ServerName documentation for more information.) In the case of FcgidInitialEnv, if two virtual hosts have the same server name but different environments as defined by FcgidInitialEnv, the environment used for a particular request will be that defined for the virtual host of the request that caused the FastCGI process to be started.

So it appears that if running multiple virtual servers on the same box its a good idea to set FcgidMaxProcessesPerClass, FcgidMinProcessesPerClass and either FcgidProcessLifetime or FcgidIdleTimeout (not sure the difference between these two) to settings that work for your particular server.. This will avoid one site using up ALL available slots (defined by FcgidMaxProcesses) and will kill off fcgid processes that are idle to reduce memory usage..

These settings can be over ridden in the VirtualHost config to allow for some fine tuning..

From my understanding, your'

From my understanding, your' saying that is not possible to use APC with fcgid because the opcode wont work ?

Thank you

Yes, and no

Yes, and no.

It will not work with one opcode shared memory segment for all the fcgid processes, but it will work with each process having its own code.

So total memory usage is more, but it does work out well in the end in most cases.

Thank you for your

Thank you for your response!

Do you have an article or example of how to do that ?

I just found this http://www.brandonturner.net/blog/2009/07/fastcgi_with_php_opcode_cache/ but is using mod_fastcgi instead mod_fcgid .

thank you for your time.

fcgid and apc

Configuring fcgid itself and Apache are covered in detail above in the article we are comment on right now.

For APC, you need the following, e.g. in /etc/php5/conf.d/apc.ini for Debian/Ubuntu:

extension = apc.so
apc.shm_size = 96
apc.rfc1867 = 1

That is all.

Thank you! I have fcgid

Thank you! I have fcgid installed, I just needed the APC part, I thought it was more dificult hehe.

Regards

Fcgid some instalation problems

Hi All,
I am trying to install Fcgid but, after installing if i try to run any application it gives me "Internal Server Error", also iam unable to find the file where i should add "Order Allow,Deny
Allow From All
Allow Override All
Options MultiViews Indexes Includes FollowSymLinks ExecCGI". Also, when i run "aptitude install apache2-mpm-worker libapache2-mod-fcgid" i get this message "The following actions will resolve these dependencies:

Remove the following packages:
1) apache2-mpm-prefork
2) libapache2-mod-php5
". Please let me know if iam doing anything wrong. Thanks in advance

Awesome post

I have recently implemented fcgid+worker to alleviate excessive memory usage which was causing server stability issues on a cPanel VPS. All sites are Drupal, as multi-site (shared core and contributed module code base). I have no stats but seems quicker than DSO. So far, so good - swapping no more.

Question: currently I have disabled SUEXEC and wondering if by enabling it there will be any negative impact to the benefits of shared code base. eg; are additional PHP processes spawned due to running SUEXEC (compiled per user) or does it work the same as without?

Hope this makes sense.

Cautiously switching from mod_php to mod_fcgid

First of all, thank you for this article, and for your many helpful posts on Drupal.org. I decided to run Drupal on my own server, and it helps to get advice from someone who (unlike me) knows what he is doing.

I run a few low-traffic sites on a small virtual server (512 MB of RAM). I probably enable more Drupal modules than I need, and so I set memory_limit to 96 MB in php.ini. As a result, I started running into trouble: too many apache processes eating up my RAM, swapping, ugh. I set MaxClients to 17 (apache prefork) and decided to switch to the worker MPM and fast CGI as a better solution.

I followed your instructions, and I think I made some improvements. I would love to know what you think. One of my goals was to make the switch cautiously, and with as little down time as possible.

First, a question. What is the meaning of the comment
#Or use this if you use the file above
in your php-fcgid.conf ?

My first step was to keep mpm-prefork but switch to mod_fcgid. I created php-fcgid.conf by copying from this article and added "Options +ExecCGI" to all my vhost files. I wrapped the apache directives for the worker mpm in <IfModule mpm_worker_module>...</IfModule> (see below) so that they would not take effect during this step. Then

su
apt-get install libapache2-mod-fcgid php5-cgi php5-curl
/etc/init.d/apache2 stop
a2dismod php5
a2enmod fcgid
/etc/init.d/apache2 start

(Did I mention that I am running Ubuntu? I upgraded to 10.04 LTS yesterday.) It all went smoothly, with almost no down time. I could have skipped the a2enmod step, because apt-get took care of that for me.

The second step was easy: switch from prefork to worker.

apt-get install apache2-mpm-worker

Again, apt-get took care of things (stopping apache and then starting it again) for me.

Now for the changes I made. First, in php-fcgid.conf, I made separate sections for the fcgid, mpm_prefork, and mpm_worker directives. Next, I checked the Apache docs and found that the fcgid directives all have new names: the ones you use still work, but they are deprecated. As a bonus, I found the documentation for the FcgidMaxProcesses directive (formerly MaxProcessCount). As a result, my php-fcgid.conf file (including some comments and an extra ServerLimit directive for the worker mpm cribbed from the main apache2.conf file) looks like this:

<IfModule fcgid_module>
AddHandler fcgid-script .fcgi .php
# Where to look for the php.ini file?
## FcgidInitialEnv PHPRC "/etc/php5/cgi"
# Maximum requests a process handles before it is terminated
# original name MaxRequestsPerProcess (now deprecated)
FcgidMaxRequestsPerProcess 1000
# Maximum number of PHP processes
# original name MaxProcessCount (now deprecated)
FcgidMaxProcesses 5
# Number of seconds of idle time before a process is terminated
# original name IPCCommTimeout (now deprecated)
FcgidIOTimeout 240
# original name IdleTimeout (now deprecated)
FcgidIdleTimeout 240
#Or use this if you use the file above
# original name FCGIWrapper (now deprecated)
FcgidWrapper /usr/local/bin/php-wrapper .php
## FcgidWrapper /usr/bin/php-cgi .php
</IfModule>

## BF: default values 5, 5, 10, 150, 0
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 4
MaxSpareServers 8
MaxClients 17
MaxRequestsPerChild 0
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
StartServers 3
MinSpareThreads 3
MaxSpareThreads 10
ThreadLimit 64
ThreadsPerChild 10
MaxClients 150
MaxRequestsPerChild 1000
ServerLimit 500
</IfModule>

I know, more comments than I need. You will notice that I commented out the PHPRC directive and that my PHP wrapper is actually a wrapper and not the cgi executable. Not a big deal, but I thought that I might also use fcgid for Perl or some other language, and they do not need the PHP environment variables.

I remembered to make the PHP wrapper executable.

One last change. Further down the page in the Apache docs, it says that one should set the environment variable PHP_FCGI_MAX_REQUESTS to be at least as large as FcgidMaxRequestsPerProcess. Since PHP_FCGI_MAX_REQUESTS defaults to 500, this might explain some of the "Fatal error" messages you mention at the end of your article. If you test this, please let us know! Again, I could have used another FcgidInitialEnv directive to set the environement variable, but since I decided to use a wrapper, I put it in there. Here is my wrapper, mostly cribbed from the URL in its first comment:

#!/bin/sh
# See http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#examples
# Set desired PHP_FCGI_* environment variables.
# Example:
# PHP FastCGI processes exit after 500 requests by default.
# Make this >= FcgidMaxRequestsPerProcess in /etc/apache2/conf.d/php-fcgid.conf
PHP_FCGI_MAX_REQUESTS=10000
export PHP_FCGI_MAX_REQUESTS
export PHPRC="/etc/php5/cgi"

# Replace with the path to your FastCGI-enabled PHP executable
exec /usr/bin/php-cgi

fcgid and mod_rewrite issues

Since switching to fcgid from suphp on apache 2.2 we have found that mod_rewrite is not working correctly for some rules - resulting in a "no input file specified" error. So far we have been unable to resolve this. Any other implementation of PHP works fine.

Should not matter

That is odd. We have not seen this, though we use such a setup for many clients.

It should not matter whether PHP is FastCGI, or mod_php or suphp.

The module mod_rewrite is an Apache module, and whether Apache is running as MPM worker or pre-fork should not impact how rewriting works.

Perhaps your rewrite rules impact PHP somehow, passing variables form Apache or something like that.

mod_fcgid: can't apply process slot for

Some times it shows this message in the error logs
mod_fcgid: can't apply process slot for /path/to/file.php
and it will show Server Busy message when accessing the website.Any one getting this error?

mod_fcgid will, in some cases, give 500 errors - by design (!)

I was getting plenty of errors in my apache error logs (about 80 a day on a high traffic site):

mod_fcgid: can't apply process slot for /usr/bin/php-cgi

These would often (but not always) show up as 503 errors in the apache access log. Running varnish in front of this setup led to a number of 503's logged in varnish from the apache backend:

FetchError c no backend connection
backend write error: 11 (Resource temporarily unavailable)
http first read error: -1 0 (No error recorded)

According to the "Special PHP considerations" paragraph at http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html

By default, PHP FastCGI processes exit after handling 500 requests, and they may exit after this module has already connected to the application and sent the next request. When that occurs, an error will be logged and 500 Internal Server Error will be returned to the client.

WTF?! 500 errors expected as part of NORMAL operations?! Apache's solution is one of 2 things:

  • Set PHP_FCGI_MAX_REQUESTS to 0 - "but that can be a problem if the PHP application leaks resources". Which means writing zombie killing scripts, or separate scripts that monitor php resources and restart when necessary.
  • "PHP_FCGI_MAX_REQUESTS can be set to a much higher value than the default to reduce the frequency of this problem. FcgidMaxRequestsPerProcess can be set to a value less than or equal to PHP_FCGI_MAX_REQUESTS to resolve the problem." Basically, this equates to "Here's how to see less errors," NOT "Here's how to fix this." Again, WTF.

I notice that 2bits doesn't get these errors, and they do seem intermittent. The drupal 6 site I've just taken over from someone else suffers from buffet binge syndrome - around 230 modules, and averages about 6 seconds for the home page to load. So what is happening is that when mod_fastcgi gets to the number of MaxRequestsPerProcess, it wants to restart that php process - but it can't because it's still busy. Then it STILL allows new php requests to use that process - except that process shouldn't be available, which leads to the "can't apply process slot for /usr/bin/php-cgi" error (at least, that's how I think this error is happening).

The solution is to start using some kind of php process manager which looks after the fastcgi php processes. Enter php-fpm.

http://php-fpm.org/ says that:

PHP-FPM (FastCGI Process Manager) is an alternative PHP FastCGI implementation with some additional features useful for sites of any size, especially busier sites.

One of its features: "Advanced process management with graceful stop/start". That's what we need!

On Ubuntu 12, here's how I installed and configured it (assuming mpm worker + fcgid is already set up).

apt-get install php5-fpm libapache2-mod-fastcgi
apt-get remove libapache2-mod-fcgid
a2dismod fcgid
a2enmod fastcgi actions alias

Edit /etc/php5/fpm/pool.d/www.conf with some sensible values (see failover.co.za post below, or leave at defaults, monitor the results, and tune), and change the following:

; Find the line below
listen = 127.0.0.1:9000
; and change it to this:
listen = /var/run/php-fpm.sock

Create /etc/apache2/conf.d/php-fastcgi.conf with these contents:


Alias /php5.fastcgi /usr/lib/cgi-bin/php5
AddHandler php-script .php
FastCGIExternalServer /usr/lib/cgi-bin/php5 -socket /var/run/php-fpm.sock
Action php-script /php5.fastcgi virtual

In /etc/php5/fpm/php.ini change any relevant directives, like max_execution_time and memory.

service apache2 restart && service php5-fpm restart

References:

Plesk

How can this work with plesk installed on the vps container?

Question to Plesk

That is a question for Plesk really.

I normally avoid these control panels, be it cPanel, Plesk, WHM, ...etc. for several reasons, such as forcing compile from source with certain combinations of version numbers, inability to use less common features, and they general stand in your way for a high performance site.

benchmarking tool?

The tutorial is great.I would just want to know about the benchmarking tool you had used to compare the performance of mod_php and fcgid. Which tool had you used for benchmarking?

Thanks

The memory usage graph is

The memory usage graph is awesome. How are you guys building it ?

Munin

Munin generates that graph.

Have you tested xcache with

Have you tested xcache with this configuration? Are there any problems? What do you think ...
I see you have created a monitor script (munin?) to plot the php-cgi processes. Can we have access to this script?
Thanks

APC vs. Xcache

APC has proven to be the most stable accelerator for newer versions of PHP. eAccelerator and Xcache experience segfaults with PHP 5.2.x sporadically.

Here is the script to allow Munin to plot how many php-cgi-processes are in use. Name it php_processes and put it in /etc/munin/plugins.


#!/bin/sh
#
# Plugin to monitor the number of PHP processes on the machine.
#
# Copyright Khalid Baheyeldin 2009 http://2bits.com
#
# Parameters:
#
# config (required)
# autoconf (optional - used by munin-config)
#
# Magick markers (optional - used by munin-config and som installation
# scripts):
#%# family=auto
#%# capabilities=autoconf

if [ "$1" = "autoconf" ]; then
echo yes
exit 0
fi

if [ "$1" = "config" ]; then
echo 'graph_title Number of php-cgi processes'
echo 'graph_args --base 1000 -l 0 '
echo 'graph_vlabel number of php-cgi processes'
echo 'graph_category apache'
echo 'graph_info This graph shows the number of php-cgi processes in the system.'
echo 'php_processes.label php-cgi'
echo 'php_processes.draw LINE2'
echo 'php_processes.info The current number of php-cgi processes.'
exit 0
fi

echo -n "php_processes.value "
find /proc -maxdepth 1 -name '[1-9]*' -exec grep php-cgi {}/cmdline \; |
grep -v grep | wc -l | sed 's/\t +//' | sed 's/ *//'

simplify munin-script for php-cgi

Simplified php-cgi munin script:

#!/bin/bash
#
# By Phil2k@gmail.com
#
if [ "$#" -gt 0 -a "$1" = "config" ]; then
echo "graph_args --base 1000 -l 0"
echo "graph_scale no"
echo "graph_vlabel number of php-cgi processes"
echo "graph_title Number of php-cgi processes"
echo "graph_category apache"
echo "graph_printf %.0lf"
echo "graph_info This graph shows the number of php-cgi processes are running."
echo "php_cgi.label php-cgi"
echo "php_cgi.draw LINE1"
echo "php_cgi.info Number of php-cgi processes."
exit 0
fi
echo "php_cgi.value "`ps -C 'php-cgi'|tail -n \+2|wc -l`

Munin PHP processes

This would be very useful for me, but the script always yields the answer 1 when I run it. This doesn't seem right!

Thanks for this

Thanks for this information.

Everything is working as expected, but I notice that the php processes are attached to apache child, so when the process dies, php-cgi will die too and APC/XCache compiled code will be lost.

Is there any workaround?

Don't think so

I don' think so.

I think the APC shared memory is shared among all the php-cgi processes, so if one exits the cache is still there for the others.

Yes, if the last one dies, then the entire APC cache is destroyed, and will be created anew when a new PHP request comes in.

Install the apc.php that came with the APC package and point your browser to it and you will see that APC is active.

Few things to watch for ....

For the sake of completeness, we have found a few wrinkles occasionally and would like people to watch out for them and give feedback as well:

1) APC will not be in memory all the time. We have not verified whether it would be shared between instances or is a per instance, although a quick investigation suggests it is shared. This means that PHP scripts have to be parsed, tokenized and re-cached more often that with mod_php. In practice this it has little negative effect on performance, but interferes with using APC as a persistent cache (e.g. performance.module summary logging, cache router, ...etc.)

2) On at least one test site, if the site does not receive requests for some time (the timeout value for fcgid?), then the first request coming in complains about insufficient memory (and says it exceeded 16MB, although PHP is configured for 128MB). Hitting F5 makes it work.

3) Long running processes get killed after the timeout expires. This can affect cron, regardless of the set_time_limit() there.

Overall, the immense memory savings outweigh all the above though.

fastcgi and PHP opcode cachers

The article "FastCGI with a PHP APC Opcode Cache" at http://www.brandonturner.net/blog/2009/07/fastcgi_with_php_opcode_cache/ describes using the 3rd-party mod_fastcgi module in place of mod_fcgi. Doing so allows you to avoid the memory overhead incurred when using a PHP opcode cacher with mod_fcgi.

Article excerpt:

"Opcode caches throw a wrench in this however, because of their inability to share the cache across FastCGI processes. Hopefully one day this will be remedied. Luckily, in the meantime, PHP is capable of playing “process manager” and a single PHP process can spawn several children to handle requests. This way the parent PHP process can instantiate the opcode cache and its children can share it. You’ll see this later when we set the PHP_FCGI_CHILDREN environment variable.

"Both mod_fcgid and mod_fastcgi can be told to limit the number of PHP processes to 1 per user. The PHP process can then be told how many children to spawn. Unfortunately mod_fcgid will only send one request per child process. The fact that PHP spawns its own children is ignored by mod_fcgid. If we use mod_fcgid with our setup, we can only handle one concurrent PHP request. This is not good. A long running request could easily block multiple smaller requests.

"mod_fastcgi will send multiple simultaneous requests to a single PHP process if the PHP process has children that can handle it. This is the reason we must use mod_fastcgi to achieve our goal of one cache per user."

Found fcgid to be safer

The issue here is stability and reliability. We found mod_fastcgi to be unreliable and stable. We found that fcgid much safer to use. At least this is true on Ubuntu 8.04 LTS Server Edition.

You set a maximum for the number of processes by putting:

MaxProcessCount 35

And the server is protected from overflowing memory and thrashing due to excessive swapping. A condition that every system administrator dreads and should plan to guard against it happening to their systems.

With the configuration in the article you linked to of -maxClassProcesses 1 and a high number of PHP_FCGI_CHILDREN, the only benefit is that APC is shared across all children, but there is no safety mechanism to guard against a high number of runaway PHP processes.

Add to that the instability that we saw with mod_fastcgi, and using fcgid was the logical choice for us.

I am a guy who was mis-leaded

I am a guy who was mis-leaded by this stupid article before.

There is nothing to compare between mod_fastcgi & mod_fcgid.
mod_fastcgid can work with php, and mod_fcgid cannot.

ok... you can use stupid way to work with php, but that is not php5.3.3.

Don't waste user's time!!

Dozens of clients disagree

mod_fcgid works for sure. This very site is running it.

There is a big difference between mod_fcgid and mod_fastcgid. The former manages the processes from outside PHP, and keeps their number in check, and is very stable. It was written specifically to overcome certain drawbacks of the latter.

We run it for dozens of clients with both PHP 5.2 (Ubuntu 8.04) and PHP 5.3 (Ubuntu 10.04), with no issues, and very good scalability. One site gets over 3 million page views per day.

Don't let your misunderstanding stand in the way of your prejudice though ...

Is your Drupal or WordPress site slow?
Is it suffering from server resources shortages?
Is it experiencing outages?
Contact us for Drupal and WordPress Performance Optimization and Tuning Consulting