Articles

A client site was facing intermittent outages, specially at peak hours.

We investigated the issue over a few days, and narrowed down the cause to certain slow queries, described in more details below.

They had tagadelic on every page, displaying a tag cloud, and from that, crawlers hit every term to paths like taxonomy/term/1581.

Slow queries causing server outage

One great feature that Drupal has is the ability to make modules run certain tasks, often heavy ones, in the background at preset intervals. This can be achieved by a module implementing hook_cron.

Core uses this feature to index new content for the search module, ping module to notify remote sites of new content, fetch new release information from drupal.org, poll other sites for RSS feeds, and more.

Various contributed modules use this for various purposes, such as mailing out newsletters, cleaning up logs, synchronizing content with other servers/sites, and much more ...

With the "Cloud" being in vogue currently, we see a lot of clients asking for cloud solutions, mostly Amazon AWS. Sadly, this is normally done without really doing a proper evaluation into whether the cost is reasonable, or the technology is suitable for their specific needs.

Amazon AWS provides some unique and compelling features. Among those are: instant provisioning of virtual servers, billing for used resources only, ability to provision more instances on demand, a wide variety of instance types, and much more.

Apache, and all other web servers, have a mechanism to write an "access log" recording every HTTP access to the server. The information that is logged is valuable, and includes things like the IP address of the user making the request, the date and time, the size of the request in bytes, the return code from the HTTP protocol, the request's URI, the referer, and the browser/operating system that the user is using.

One of the things that a default Drupal installation does is that it handles 404s for static files within Drupal itself. In other words, a 404 for a .css or .jpg file causes a full Drupal bootstrap. This is not necessary, and wastes resources on a site that gets lots of 404s: lots of code is executed, many database queries are performed, only to return a 404 for a static file.

The latest Long Term Support (LTS) release from Ubuntu has been with us for a bit over a month now, and I love it so far, on my laptop. This is Lucid Lynx 10.04, and will be supported until 2013 on the desktop, and more importantly, until 2015 on the server.

My servers are still on 8.04 though, the previous LTS, for good reason. 10.04 has PHP 5.3. While this is generally a better PHP, it has some compatibility issues with many Drupal, mainly for contributed modules, but some in core as well.

The long term solution to this is to find all PHP 5.3 compatibility issues and fix them in the code. This is a lengthy process though, and will not happen overnight.

So, in the meantime, there are several solutions for running PHP 5.2 on Lucid, some better than others.

In this post, we examine the different approaches, and the merits of each.

In a previous article, we showed how a combination of Microsoft's WebDAV MiniReDir and Drupal Single SignOn result in an unintentional aggressive crawler that has the same effect as a Denial of Service attack (DoS).

And in our session yesterday at DrupalCon we touched on that topic again.

So, in this article we expand on this topic and provide more details.

The session that I proposed for DrupalCon San Francisco 2010 on 2.4 million page views per day, 60 million pages per month, one server was accepted, and I just finished giving it.

Note that from the time we proposed the talk, until we actually delivered it, the site hit new records (2.8 million page views per day, 70 million page views per month).

Here are the slides from the session, which I designed to be useful on their own, even for those who did not attend.

One of the nice things about Drupal is its pluggable caching system. It allows site owners to choose from a vast array of caching backends, such as memcache, APC, files, as well as others.

The memcache module is one of the most popular of these modules, providing Drupal integration with memcached, and in-memory distributed object caching daemon.

All Drupal pluggable caching modules work by providing a caching .inc file that is added in the settings.php for the site like this:

$conf['cache_inc'] = './sites/all/modules/memcache/memcache.inc';

Because of this pluggability, you can can effectively extend the caching in creative ways.

For high traffic Drupal sites, using the memcache module is a common method for making sites scale. The memcache module uses the memcached object caching daemon for doing the actual caching.

In this article, we document how we ran into a limitation of memcached, and how we overcame it by a simple patch.