Benchmarking Drupal with PHP op-code caches: APC, eAccelerator and XCache compared

Almost every one knows that running high traffic sites powered with PHP applications is impossible without a PHP op-code cache / accelerators. Op-code caches speed up PHP applications by parsing and tokenizing PHP scripts once, and executing them faster for every subsequent request.

There are several accelerators available, many of them are free. The ones that are maintained are:

• APC
• eAccelerator
• XCache

Sometime ago, we conducted a benchmark of Drupal using APC vs eAccelerator where the latter was found to be a better performer than the former in terms of speed and memory consumption.

It is now time to update this benchmark and include XCache as well.

Configuration

The tests were run on the following configuration:

Hardware

AMD Athlon 64 X2 Dual Core Processor 4400+ @ 2.2GHz, 1MB cache and 2GB RAM. 160GB SATA 7200RPM hard disk.

Software

GNU/Linux Ubuntu server Gutsy 7.10, Apache 2.2.4, MySQL 5.0.45 and PHP 5.2.3.

Drupal

The tests were run on Drupal 6.3-dev, checked out from CVS. Views and CCK are installed and enabled, but there are no views nor CCK types defined.

Using the devel generate module, we created 2,000 users, 5 vocabularies, and 50 terms. We then created 2,000 nodes of type page and story, with 5 comments each, and assigned terms to them.

All Drupal caching is disabled.

PHP op-code caches

We used the following versions of each op-code cache, we also note how it was installed, and what the configuration details we used:

XCache

XCache is a spinoff of the lighttpd web server. It is currently maintained.

We used version: 1.2.1, which is available from the APT repositories for Ubuntu Gutsy, as php5-xcache.

Being available in the Ubuntu/Debian repositories means that installing Xcache was the easiest of the bunch using aptitude.

Configuring XCache requires some parameters to change though if you want the cache to be in memory though. We used the following configuration to change the file which resides in /etc/php5/conf.d/xcache.ini.

[xcache]
xcache.shm_scheme =        "mmap"
xcache.size  =                48M
xcache.count =                 2
xcache.slots =                8K
xcache.ttl   =                 0
xcache.gc_interval =           0
xcache.readonly_protection = Off
xcache.mmap_path =           "/var/cache/xcache.mmap"
xcache.coredump_directory =  ""
xcache.cacher =              On
xcache.stat   =              On
xcache.optimizer =           Off
xcache.var_size  =            0M
xcache.var_count =             1
xcache.var_slots =            8K
xcache.var_ttl   =             0
xcache.var_maxttl   =          0
xcache.var_gc_interval =     300
xcache.test =                Off 

eAccelerator

eAccelerator is a fork of an older op-code cache called Turck MMCache that has been abandoned. eAccelerator development seems to have been slow of late, and is not keeping up with the latest versions of PHP 5. We found that eAccelerators can be unstable with newer versions of PHP and Drupal, and Apache will die with segmentation faults often. We had to use the log watcher script for restarting Apache automatically when this happens, but it was too frequent in some cases to be a real nuisance.

We used version v0.9.5.2, which is the latest stable release.

It was installed from source using the following commands:

phpize
./configure
make
make install 

We used the following configuration which resides in /etc/php5/conf.d/eaccelerator.ini.

zend_extension                  = /usr/lib/php5/20060613/eaccelerator.so
eaccelerator.shm_size           = 48
eaccelerator.cache_dir          = /var/cache/eaccelerator
eaccelerator.enable             = 1
eaccelerator.optimizer          = 1
eaccelerator.check_mtime        = 0
eaccelerator.debug              = 0
eaccelerator.filter             = ""
eaccelerator.shm_max            = 0
eaccelerator.shm_ttl            = 0
eaccelerator.shm_prune_period   = 0
eaccelerator.shm_only           = 1
eaccelerator.compress           = 1
eaccelerator.compress_level     = 9  

APC

APC is a PECL package that is maintained by the core PHP developers, including Gopal and Rasmus. It has several advantages including a very simple configuration, and close tracking of PHP versions. Being actively maintained is a big plus for APC.

We used version 3.0.16 of APC. It was installed using PECL, via the following command:

pecl install apc 

The configuration for APC is very minimalistic. We used the following configuration file in /etc/php5/conf.d/apc.ini.

extension                 = apc.so
apc.shm_size              = 48

Benchmarking methodology

We used the Apache Benchmark (ab) command, with a concurrency of 5, and 3,000 requests, like so:

ab -c5 -n3000 http://example.com/ 

Test 1: No PHP op-code cache

In this test, we established a baseline of how Drupal 6 would perform without any op-code cache.

Document Path:          /
Document Length:        21757 bytes
Concurrency Level:      5
Time taken for tests:   288.255212 seconds
Complete requests:      3000
Failed requests:        0
Write errors:           0
Total transferred:      66777000 bytes
HTML transferred:       65271000 bytes
Requests per second:    10.41 [#/sec] (mean)
Time per request:       480.425 [ms] (mean)
Time per request:       96.085 [ms] (mean, across all concurrent requests)
Transfer rate:          226.23 [Kbytes/sec] received
Connection Times (ms)
min  mean[+/-sd] median   max
Connect:        0    0   0.5      0      19
Processing:   181  479 186.0    444    1822
Waiting:      166  461 184.7    427    1708
Total:        181  479 186.0    444    1822
Percentage of the requests served within a certain time (ms)
50%    444
66%    525
75%    577
80%    619
90%    732
95%    819
98%    946
99%   1012
100%   1822 (longest request) 

Using the devel module, we see that the page generation time for PHP is around 200 ms.

Page execution time was 209.58 ms. Executed 101 queries in 9.6 milliseconds. 

Memory utilization is consistent at 24MB per Apache process. Note that the sixth colum (Resident Set Size) is what matters.

                         Virt Res
13616 www-data  16   0  213M 24660  3744 S  0.0  1.2  0:51.37 /usr/sbin/apache2 -k start
13619 www-data  15   0  213M 24484  3824 S  0.0  1.2  0:51.37 /usr/sbin/apache2 -k start
13627 www-data  15   0  213M 24484  3824 S  0.0  1.2  0:50.95 /usr/sbin/apache2 -k start
13617 www-data  15   0  213M 24484  3824 S  0.0  1.2  0:50.14 /usr/sbin/apache2 -k start
13624 www-data  15   0  213M 24408  3748 S  0.0  1.2  0:48.99 /usr/sbin/apache2 -k start
13626 www-data  16   0  213M 24408  3748 S  0.0  1.2  0:50.07 /usr/sbin/apache2 -k start
13615 www-data  16   0  213M 24404  3744 S  0.0  1.2  0:51.03 /usr/sbin/apache2 -k start
13623 www-data  15   0  213M 24404  3744 S  0.0  1.2  0:49.39 /usr/sbin/apache2 -k start
13625 www-data  15   0  213M 24404  3744 S  0.0  1.2  0:52.95 /usr/sbin/apache2 -k start
13618 www-data  16   0  213M 24404  3744 S  0.0  1.2  0:49.93 /usr/sbin/apache2 -k start 

Test 2: eAccelerator

Document Path:          /
Document Length:        21757 bytes
Concurrency Level:      5
Time taken for tests:   95.983986 seconds
Complete requests:      3000
Failed requests:        0
Write errors:           0
Total transferred:      66777000 bytes
HTML transferred:       65271000 bytes
Requests per second:    31.26 [#/sec] (mean)
Time per request:       159.973 [ms] (mean)
Time per request:       31.995 [ms] (mean, across all concurrent requests)
Transfer rate:          679.39 [Kbytes/sec] received
Connection Times (ms)
min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       3
Processing:    57  159  91.3    148    3830
Waiting:       50  152  89.8    142    3704
Total:         57  159  91.3    148    3830
Percentage of the requests served within a certain time (ms)
50%    148
66%    174
75%    193
80%    205
90%    239
95%    263
98%    289
99%    309
100%   3830 (longest request)

Using devel, the PHP time is around 47 ms. A significant improvement over not using an op-code cache.

Page execution time was 57.88 ms. Executed 101 queries in 9.01 milliseconds. 

Memory utilization for Apache is as follows. The 30MB process size is from the first run where the PHP scripts were loaded, parsed and tokenized. The other processes are from the subsequent requests and they range from 23MB to 18MB.

                         Virt Res
9801 www-data  16   0  261M 30688 12644 S  0.0  1.5  0:14.96 /usr/sbin/apache2 -k start
9799 www-data  16   0  254M 23848 12588 S  0.0  1.2  0:15.89 /usr/sbin/apache2 -k start
9797 www-data  16   0  253M 22536 12272 S  0.0  1.1  0:15.74 /usr/sbin/apache2 -k start
9800 www-data  15   0  251M 20028 11832 S  0.0  1.0  0:14.08 /usr/sbin/apache2 -k start
9806 www-data  16   0  251M 18536 10396 S  0.0  0.9  0:14.74 /usr/sbin/apache2 -k start
9808 www-data  16   0  251M 18536 10396 S  0.0  0.9  0:14.83 /usr/sbin/apache2 -k start
9807 www-data  15   0  251M 18536 10396 S  0.0  0.9  0:14.67 /usr/sbin/apache2 -k start
9803 www-data  15   0  251M 18536 10396 S  0.0  0.9  0:15.62 /usr/sbin/apache2 -k start
9809 www-data  16   0  251M 18536 10396 S  0.0  0.9  0:14.07 /usr/sbin/apache2 -k start
9805 www-data  16   0  251M 18536 10396 S  0.0  0.9  0:14.16 /usr/sbin/apache2 -k start

Test 3: XCache

Document Path:          /
Document Length:        21757 bytes
Concurrency Level:      5
Time taken for tests:   99.76300 seconds
Complete requests:      3000
Failed requests:        0
Write errors:           0
Total transferred:      66777000 bytes
HTML transferred:       65271000 bytes
Requests per second:    30.28 [#/sec] (mean)
Time per request:       165.127 [ms] (mean)
Time per request:       33.025 [ms] (mean, across all concurrent requests)
Transfer rate:          658.19 [Kbytes/sec] received
Connection Times (ms)
min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       2
Processing:    59  164  83.4    155    3367
Waiting:       52  156  66.4    148    1802
Total:         59  164  83.4    155    3367
Percentage of the requests served within a certain time (ms)
50%    155
66%    178
75%    196
80%    206
90%    237
95%    263
98%    287
99%    305
100%   3367 (longest request) 

Using devel, the page generation is around 50 ms.

 Page execution time was 59.37 ms. Executed 101 queries in 9.27 milliseconds.

For memory utilization, you can see that it ranges from 29MB to 19MB, a bit more than eAccelerator.

                         Virt Res
10316 www-data  16   0  263M 32316 11832 S  0.0  1.6  0:17.03 /usr/sbin/apache2 -k start
10319 www-data  15   0  259M 29976 13440 S  0.0  1.5  0:17.05 /usr/sbin/apache2 -k start
10318 www-data  15   0  261M 29724 11628 S  0.0  1.4  0:16.83 /usr/sbin/apache2 -k start
10317 www-data  16   0  254M 23576 12128 S  0.0  1.1  0:15.33 /usr/sbin/apache2 -k start
10322 www-data  15   0  251M 19624 11524 S  0.0  1.0  0:16.85 /usr/sbin/apache2 -k start
10328 www-data  15   0  251M 19624 11524 S  0.0  1.0  0:14.95 /usr/sbin/apache2 -k start
10324 www-data  15   0  251M 19624 11524 S  0.0  1.0  0:15.32 /usr/sbin/apache2 -k start
10325 www-data  15   0  251M 19624 11524 S  0.0  1.0  0:14.42 /usr/sbin/apache2 -k start
10327 www-data  16   0  251M 19624 11524 S  0.0  1.0  0:15.05 /usr/sbin/apache2 -k start

Test 4: APC

Document Path:          /
Document Length:        21757 bytes
Concurrency Level:      5
Time taken for tests:   98.530068 seconds
Complete requests:      3000
Failed requests:        0
Write errors:           0
Total transferred:      66777000 bytes
HTML transferred:       65271000 bytes
Requests per second:    30.45 [#/sec] (mean)
Time per request:       164.217 [ms] (mean)
Time per request:       32.843 [ms] (mean, across all concurrent requests)
Transfer rate:          661.84 [Kbytes/sec] received
Connection Times (ms)
min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       2
Processing:    58  163  71.2    155    2452
Waiting:       53  158  69.6    150    2329
Total:         58  163  71.2    155    2452
Percentage of the requests served within a certain time (ms)
50%    155
66%    178
75%    193
80%    204
90%    235
95%    258
98%    285
99%    302
100%   2452 (longest request) 

Using devel, the page generation time is around 50 ms as well.

Page execution time was 59.8 ms. Executed 101 queries in 9.1 milliseconds. 

Memory utilization is noticeably consistent at 21MB per process, more than the other two op-caches, but suprisingly consistent.

                        Virt Res
9263 www-data  16   0  263M 38172 18036 S  0.0  1.9  0:15.31 /usr/sbin/apache2 -k start
9266 www-data  15   0  252M 21704 13020 S  0.0  1.1  0:18.14 /usr/sbin/apache2 -k start
9270 www-data  15   0  252M 21604 12768 S  0.0  1.0  0:18.30 /usr/sbin/apache2 -k start
9274 www-data  15   0  252M 21604 12768 S  0.0  1.0  0:15.75 /usr/sbin/apache2 -k start
9264 www-data  16   0  252M 21520 12692 S  0.0  1.0  0:16.55 /usr/sbin/apache2 -k start
9267 www-data  15   0  252M 21520 12688 S  0.0  1.0  0:18.10 /usr/sbin/apache2 -k start
9268 www-data  16   0  252M 21520 12688 S  0.0  1.0  0:16.89 /usr/sbin/apache2 -k start
9269 www-data  15   0  252M 21520 12688 S  0.0  1.0  0:16.51 /usr/sbin/apache2 -k start
9273 www-data  16   0  252M 21520 12688 S  0.0  1.0  0:17.32 /usr/sbin/apache2 -k start
9275 www-data  16   0  252M 21520 12688 S  0.0  1.0  0:16.03 /usr/sbin/apache2 -k start 

Summary

The following table summarizes the above results.

Conclusions

From the above results, one can come to the following conclusions:

• All op-code caches provide a noticable improvement for Drupal over a default PHP installation.
• The speed gain is about 3X.
• eAccelerator is marginally better than the XCache or APC both in terms of speed and memory utilization.
• Installation of each op-code cache is different: one has a Debian package, the other is installed from source and the third is via PECL.
• The configuration for each is also different. Some work well with a default install, others require more tweaking.

Update:

We have noticed that in production, Xcache suffers from the same instability that eAccelerator exhibits: segmentation fault after a day or so. This happened with Xcache 1.2.1-3 which ships with Ubuntu 8.04.1, and PHP 5.2.4. The latest stable release is 1.2.2 from Xcache's web site. So, try compiling that from source, or install the logwatcher script if a minute of downtime is acceptable.