Installing memcached 1.2.5 and memcache pecl 2.2.3 on Ubuntu server

A large web site that 2bits.com built and manages gets over 1 million page views a day. It has been using Drupal's Memcached API module for several months. The version installed until yesterday was 5.x-1.1. Since there is a newer version (5.x-1.7), it was time to upgrade.

The upgrade seemed to have gone well, but we saw the load on the CPU to go up a lot during peak hours.

Also the load on the database went up considerably:

As you can see from the graph, there was a lot of insert activity going on.

This article explains how we identified the issue, and how we solved it by installing and configuring the latest version of memcached and PECL's memcache. It was done on a live server running Ubuntu 6.10, but it applies to 7.10 as well. The exact versions may change when 8.04 is released later this month.

Investigating the problem

Checking the watchdog log, there were lots of errors like this one:

Failed to set key: cache-variables

Checking the code, we found this snippet was due to this code in dmemcache.inc:

if (!$mc->set($full_key, $value, MEMCACHE_COMPRESSED, $exp)) {
watchdog('memcache', 'Failed to set key: ' . $full_key, WATCHDOG_ERROR);
}

The project page for the memcache module states that we should use PECL 2.2.1 or later.

Even on Ubuntu Gutsy 7.10, the PECL memcache package is 2.1.2, so we have to go the more arduous route of installing via pecl, and compiling memcached from source.

Installing memcache pecl

First, we need to install the libevent dev library, so when we install pecl memcache, it will compile correctly.

To do this, we run the following command:

aptitude install libevent-dev

Now we are ready to install pecl memcache, which happens to be 2.2.3 at the time of this article.

pecl install memcache

All goes well, and the memcache.so library is built.

We do not need to change the php.ini, because the entries required in it were installed by the older version of the php5-memcache package.

Installing memcacehd from source

We can now proceed to install memcached from source, since the 2.2.3 pecl memcache will not work with the older version provided by Ubuntu's repository for this distro version.

We first download the latest version of memcached:

wget http://www.danga.com/memcached/dist/memcached-1.2.5.tar.gz

We then extract it, and change to the directory:

tar xzvf memcached-1.2.5.tar.gz
cd memcached-1.2.5

The following commands cause it to be build from source:

./configure
make
make install

Now we need to stop the old memached:

/etc/init.d/memcached stop

And also stop Apache:

/etc/init.d/apache2 stop

In order to prevent the old version that is supplied with the distro from starting again, we issue this command:

update-rc.d -f memcached remove

Then we start the new memcache manually using the following command:

/usr/local/bin/memcached -d -u root -m 512 -p 11211

Then we start apache

/etc/init.d/apache2 start

And all is well. Memcache is now caching our stuff, instead of it being cached in the database.

You can see the improvement in the above graphs where the CPU and database load went down significantly.

But we need to do some house cleaning ...

First we create a script in /usr/local/bin/memcache.sh script

#!/bin/sh
case "$1" in
start) /usr/local/bin/memcached -d -u root -m 512  -p 11211
;;
stop)  killall memcached
;;
esac 

Note that this is a default install with all of the caches in one bin.

You can fine tune the process further by using multiple bins each with different size by observing memcache for a week or so, and adjusting the values. Start with a large enough value for each bin (e.g. 256MB), and then tune down from there. Check our followup article on configuration for Drupal and memcache for details.

Then we edit the /etc/rc.local file to add this to it:

/usr/local/bin/memcached start

This way, we don't have to start it manually if the server is rebooted.

There is one more thing to do: that is to investigate the Cache Router module, which supercedes the memcache module as well as other caching modules.