Drupal site excessive memory usage traced to APC and php5-memcached

We were recently troubleshooting a client site running Drupal 7.x, and main complaint was high memory usage on all the site's pages.

We were able to diagnose and solve two main causes that range from the common to unusual.

This is a Drupal 7 Commerce site with 173 modules, and 3 themes enabled. Apache Solr is used for search, and there is custom code to talk over the network to a non-Drupal backend server.

The site runs on a Debian Squeeze Xen VPS.

For most of the site's pages, the client was seeing high memory usage, as follows:

Problem: high memory usage

When every page load has extremely excessive memory usage, this could be a bottleneck for the site scaling well, since the server has to have enough memory to cope with many pages at the same time using lots of memory.

The first access to a page, where APC has not yet cached anything, would look like this in devel:

Memory used at: devel_boot()=6.92 MB, devel_shutdown()=236.03 MB, PHP peak=243.25 MB.

Subsequent access would show less memory usage, since APC caches the Drupal PHP files, like so:

Memory used at: devel_boot()=6.54 MB, devel_shutdown()=174.37 MB, PHP peak=175.5 MB.

Some page even reported up to 192 MB of peak memory!

That is still excessive. For a site with that many modules, we expected that memory usage would be high, but not to that extent.

Solutions to high memory usage

Increasing APC shared memory size

First, the allocated shared memory for APC was not enough.

The apc.shm_size parameter for APC is set to the default of 32MB.

The code base with that many modules needed at least double that or more.

So, increasing this to 96MB, solved that part.

To do so on Debian or Ubuntu, change the following line in the file /etc/php5/apache2/conf.d/apc.ini

apc.shm_size = 96

Replacing php5-memcached with php-memcache

The other big cause of excessive memory usage was quite unusual. It was the use of php5-memcached (notice the "d") to connect PHP with the memcached daemon, rather than the more commonly used php5-memcache (without a "d").

For some unknown reason, the PHP memcached extension (from the Debian package php5-memcached) uses way more memory than the php5-memcache extension.

In order to remedy this, do the following:

$ sudo aptitude purge php5-memcached
$ sudo aptitude install php5-memcache

What a difference a "d" makes!

Results

The results after doing both of the above things were dramatic. Instead of 175 MB per page, it is now a more typical (for a complex site): 60 MB!

Memory used at: devel_boot()=2.15 MB, devel_shutdown()=58.4 MB, PHP peak=59.5 MB.

Note that these figures are not absolute, and will vary from distro to distro and server to server, depending on what modules you have enabled in PHP and Apache, and many other factors. What matters is the comparative figures, not absolute figures.

For example, the same site on an Ubuntu Server LTS 12.04, which we used for our lab servers:

Memory used at: devel_boot()=2.11 MB, devel_shutdown()=72.27 MB, PHP peak=75.75 MB.

It will be different on CentOS.