Server indigestion: The Drupal contributed modules "open buffet binge" syndrome

Often times, I call contributed modules the "open buffet" of Drupal. As opposed to a la carte. In an open buffet, you pay a fixed price, and eat all you can. In an a la carte restaurant, you order each item and pay for them separately.

Each method has its pros and cons, but today we will discuss how an open buffet can cause binge eating, indigestion and other undesirable consequences.

We compare two sites. One is minimal Drupal install, with a few modules enabled, and another bloated site with too many modules.

The methodology is to restart Apache, and do one page refresh for each site, and measure Apache's resident set size (RES) using the top command, sorted by the RES column.

Note that we have discussed apache2's bloat from Ubuntu Edgy (6.10) to Ubuntu Feisty (7.04) in a previous article.

We start with Apache just restarted, and we see the following:

Each apache process is a little above 5MB.

top - 22:30:45 up 65 days, 10:07, 17 users,  load average: 0.25, 0.53, 0.41
Tasks: 123 total,   2 running, 121 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.0%sy,  0.0%ni, 99.6%id,  0.0%wa,  0.2%hi,  0.0%si,  0.0%st
Mem:   1027356k total,   339388k used,   687968k free,     8508k buffers
Swap:  3903712k total,    85004k used,  3818708k free,   243212k cached
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16324 mysql     15   0  230m  14m 3760 S  0.0  1.4  17:59.27 mysqld
9770 root      15   0  219m 9292 4540 S  0.0  0.9   0:00.05 apache2
9772 www-data  23   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9773 www-data  23   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9774 www-data  25   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9775 www-data  25   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9776 www-data  25   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2

Now we load a basic Drupal 5.x site, that only has a few modules enabled:

mysql> select name from system 
where type = 'module' and status = 1 
order by name;
+----------------+
| name           |
+----------------+
| block          |
| color          |
| comment        |
| devel          |
| devel_generate |
| filter         |
| help           |
| menu           |
| node           |
| system         |
| taxonomy       |
| tracker        |
| user           |
| watchdog       |
+----------------+

14 modules enabled in total.

After one page load of that site's home page (/node).

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
9772 www-data  15   0  233m  33m  15m S  0.0  3.4   0:00.58 apache2
16324 mysql     15   0  230m  14m 3760 S  0.0  1.4  17:59.28 mysqld
9770 root      15   0  219m 9292 4540 S  0.0  0.9   0:00.05 apache2
9773 www-data  15   0  219m 6164 1336 S  0.0  0.6   0:00.01 apache2
9774 www-data  25   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9775 www-data  25   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9776 www-data  25   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9778 www-data  15   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9779 www-data  15   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9780 www-data  15   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2 

Note that one apache process has increased in size, from 5MB to 33MB.

After several reloads of the home page:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
9772 www-data  16   0  233m  33m  15m S  0.0  3.4   0:00.76 apache2
9774 www-data  15   0  225m  20m  10m S  0.0  2.1   0:01.27 apache2
9775 www-data  15   0  225m  20m  10m S  0.0  2.1   0:01.30 apache2
9776 www-data  15   0  223m  18m  10m S  0.0  1.9   0:00.56 apache2
9778 www-data  15   0  223m  18m  10m S  0.0  1.8   0:00.34 apache2
9773 www-data  15   0  223m  17m  10m S  0.0  1.8   0:00.17 apache2
9780 www-data  16   0  223m  17m  10m S  0.0  1.8   0:00.16 apache2
9779 www-data  16   0  223m  17m  10m S  0.0  1.8   0:00.24 apache2
16324 mysql     15   0  230m  14m 3760 S  0.0  1.4  17:59.52 mysqld
9770 root      15   0  219m 9292 4540 S  0.0  0.9   0:00.05 apache2
9782 www-data  15   0  219m 6164 1336 S  0.0  0.6   0:00.00 apache2

So, the size of an apache process is 17 to 33 MB. Not optimal, but still acceptable.

On a server with 500MB, you can fit at least 15 Apache processes in memory.

We now conduct the same tests on another site, where the "open buffet binge syndrome" is evident:

This one has 113 modules enabled, as follows:

mysql> select name from system 
where type = 'module' and status = 1 
order by name;
+-------------------------------+
| name                          |
+-------------------------------+
| actions                       |
| activeselect                  |
| ad                            |
| adsense                       |
| adsense_injector              |
| advuser                       |
| ad_cache_file                 |
| ad_embed                      |
| ad_image                      |
| ad_notify                     |
| ad_report                     |
| ad_text                       |
| aggregator                    |
| akismet                       |
| basicevent                    |
| block                         |
| blog                          |
| calendar                      |
| calendar_ical                 |
| captcha                       |
| cck_field_perms               |
| color                         |
| content                       |
| content_taxonomy              |
| content_taxonomy_activeselect |
| content_taxonomy_autocomplete |
| content_taxonomy_options      |
| creativecommons_lite          |
| custom                        |
| custom_links                  |
| date                          |
| date_api                      |
| dblclick                      |
| devel                         |
| editview                      |
| email                         |
| event                         |
| eventrepeat                   |
| eventrepeat_views             |
| event_all_day                 |
| event_views                   |
| fckeditor                     |
| feedfield                     |
| fieldgroup                    |
| filter                        |
| fivestar                      |
| flag_content                  |
| form_collect                  |
| form_store                    |
| gmap                          |
| gmap_cck                      |
| gmap_location                 |
| gmap_macro_builder            |
| gmap_views                    |
| googleanalytics               |
| help                          |
| imagefield                    |
| insert_view                   |
| jscalendar                    |
| jstools                       |
| link                          |
| location                      |
| location_cookie               |
| location_views                |
| masquerade                    |
| menu                          |
| mimemail                      |
| node                          |
| nodecomment                   |
| nodefamily                    |
| nodequeue                     |
| nodereference                 |
| number                        |
| optionwidgets                 |
| pageroute                     |
| pageroute_nodefamily          |
| pageroute_ui                  |
| panels                        |
| path                          |
| pathauto                      |
| planet                        |
| profile                       |
| scheduler                     |
| search                        |
| send                          |
| statistics                    |
| subform_element               |
| system                        |
| taxonomy                      |
| taxonomySearch                |
| taxonomy_fields               |
| text                          |
| tinymce                       |
| tracker                       |
| update_status                 |
| upload                        |
| user                          |
| userreference                 |
| viewfield                     |
| views                         |
| views_argument_api            |
| views_bonus                   |
| views_bookmark                |
| views_fastsearch              |
| views_filterblock             |
| views_fusion                  |
| views_multiblock              |
| views_rss                     |
| views_theme_wizard            |
| views_ui                      |
| votingapi                     |
| watchdog                      |
| webform                       |
+-------------------------------+
113 rows in set (0.00 sec)

After a single page refresh to the home page, we see the following.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
9797 www-data  15   0  275m  93m  33m S  0.0  9.4   0:03.80 apache2
16324 mysql     15   0  230m  14m 3764 S  0.0  1.5  17:59.68 mysqld
9793 root      15   0  219m 9292 4540 S  0.0  0.9   0:00.05 apache2
9798 www-data  15   0  220m 6364 1332 S  0.0  0.6   0:00.02 apache2
9795 www-data  15   0  219m 6116 1296 S  0.0  0.6   0:00.00 apache2
9796 www-data  15   0  219m 6116 1296 S  0.0  0.6   0:00.00 apache2
9799 www-data  15   0  219m 5968 1148 S  0.0  0.6   0:00.00 apache2
10314 www-data  15   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
10315 www-data  15   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2

We see that the size of an Apache process is 93M (as opposed to 33M for a site with less modules).

After several page refreshes, things are as follows:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
9795 www-data  15   0  279m  98m  33m S  0.0  9.8   0:06.74 apache2
9799 www-data  15   0  279m  98m  33m S  0.0  9.8   0:06.52 apache2
9797 www-data  15   0  275m  93m  33m S  0.0  9.4   0:03.81 apache2
10314 www-data  15   0  275m  93m  33m S  0.0  9.4   0:03.73 apache2
10315 www-data  15   0  275m  93m  33m S  0.0  9.4   0:03.67 apache2
9798 www-data  15   0  275m  92m  34m S  0.0  9.2   0:04.24 apache2
16324 mysql     15   0  230m  15m 3880 S  0.0  1.5  18:00.42 mysqld
9793 root      15   0  219m 9292 4540 S  0.0  0.9   0:00.05 apache2
9796 www-data  15   0  220m 6440 1408 S  0.0  0.6   0:00.01 apache2 

The size of an apache process is 92 to 98 MB. That is way too much.

On a server with 500MB of available RAM, you can barely fit 5 processes at the same time in memory. Better have Apache's MaxClients tuned to 5 otherwise , if you get traffic spikes, your server will go to thrashing hell.

Don't even think of running such a site on shared hosting with medium traffic. Your host will quickly terminate your account for overuse of resources.

So far, full featured sites seem to have about 80 to 110 modules. I have seen a case where a site had 122 module.

Keeping the number of modules down can help a lot. See our article on whether a Drupal site can handle a million page views a day.

Update July 2010:

Check out our presentation on a site that does 2.8 million page views per day, 70 million per month, one server!.

That site broke the record later, and did 3.4 million page views in one day, and 92 million in one month.

Update September 2010:

We constantly see clients that go for an insanely large amount of modules. For example, we have seen several with 180 or so modules.

But it even goes beyond our worst nightmare. I once woke up from sleep with a nightmare that a client called us to help with a site that had 200 modules.

Then it was in real life that a client called about a site with 228 modules, and another had 231 modules! Later another one with 238 engaged us in consulting for that poor site.

Update January 2011

Here is a real life case of the effect of the number of modules on importing of new nodes. David Kent Norman has an article detailing how he made things much more speedy by disabling modules.

Update October 2012

New world record for number of enabled modules: 381 modules! More details at Presentation: Huge! Drupal site with 381 modules, 174 GB MySQL database and tables over 200 million rows. The good news is that we were able to get that down to 207 modules only. It can be done!

Contents: 

Comments

Definitely feeling the pain

We have a similarly-sized setup with 112 enabled modules and a total Drupal core + modules code base size of ~500,000 lines, and are indeed having some little trouble with performance :-) Code loading sure could be a tad more fine-grained than it is at present... paying for unused features hurts.

Drupal 6, op-code cache, and mind set

Drupal 6 has several modules that are split into admin and user functions, so we don't load everything all the time.

However, with a PHP op-code cache, your code is compiled once and cached, and the overhead of loading/parsing/compiling is not there. The tests above were all conducted with APC enabled. Can't even think how things would be much worse with APC off!

What is there is that hooks that get fired for each module, allocating data structures, ...etc.

My post was mainly towards the end user mind set where they "oh, there is a module that does X? Let us have it". Apart from bloat, there is the ongoing care and feeding of modules. If you want to upgrade to the next version and 20 modules are not yet available, you either stay behind or pay someone to upgrade those modules.

So, it is a problem from many angles.

One would hope that the features provided by those extra modules are indeed useful, and worth those problems from a site functionality point of view.
--
2bits -- Drupal consulting

Devel module

Are the numbers output from the Devel module equivalent, or do they only include the PHP process, not the RAM Apache uses?
I can't seem to get the same columns from top, so I'm not sure if I'm comparing like for like.
Very useful post though - I'd never known quite how to go about deciding on the 'MaxClients' setting, thanks!

I do not think your high

I do not think your high memory usage has most to do with module's code. Take a look at this article that explains process shared memory and data: http://www.kdedevelopers.org/node/1445

If by this rule of thumb we take a difference between RES and SHR this would come to each process allocating about 60M to data. This could be caused by just a few (even one!) memory hogger / leaker modules, not necessarily the number of modules itself. I have no idea which one is prime suspect on your list.

I guess we need to think of better memory profiling tools.

This is nothing. I inherited

This is nothing. I inherited a D6 site with 310 modules that I need to put on a diet. It's FrankenDrupal!

What about custom pager? How

What about custom pager? How do they compare?

http://drupal.org/project/custom_pagers

New world record

We found a site that holds a new record: 381 enabled modules!