Server indigestion: The Drupal contributed modules "open buffet binge" syndrome

Often times, I call contributed modules the "open buffet" of Drupal. As opposed to a la carte. In an open buffet, you pay a fixed price, and eat all you can. In an a la carte restaurant, you order each item and pay for them separately.

Each method has its pros and cons, but today we will discuss how an open buffet can cause binge eating, indigestion and other undesirable consequences.

We compare two sites. One is minimal Drupal install, with a few modules enabled, and another bloated site with too many modules.

The methodology is to restart Apache, and do one page refresh for each site, and measure Apache's resident set size (RES) using the top command, sorted by the RES column.

Note that we have discussed apache2's bloat from Ubuntu Edgy (6.10) to Ubuntu Feisty (7.04) in a previous article.

We start with Apache just restarted, and we see the following:

Each apache process is a little above 5MB.

top - 22:30:45 up 65 days, 10:07, 17 users,  load average: 0.25, 0.53, 0.41
Tasks: 123 total,   2 running, 121 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.0%sy,  0.0%ni, 99.6%id,  0.0%wa,  0.2%hi,  0.0%si,  0.0%st
Mem:   1027356k total,   339388k used,   687968k free,     8508k buffers
Swap:  3903712k total,    85004k used,  3818708k free,   243212k cached
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
16324 mysql     15   0  230m  14m 3760 S  0.0  1.4  17:59.27 mysqld
9770 root      15   0  219m 9292 4540 S  0.0  0.9   0:00.05 apache2
9772 www-data  23   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9773 www-data  23   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9774 www-data  25   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9775 www-data  25   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9776 www-data  25   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2

Now we load a basic Drupal 5.x site, that only has a few modules enabled:

mysql> select name from system 
where type = 'module' and status = 1 
order by name;
+----------------+
| name           |
+----------------+
| block          |
| color          |
| comment        |
| devel          |
| devel_generate |
| filter         |
| help           |
| menu           |
| node           |
| system         |
| taxonomy       |
| tracker        |
| user           |
| watchdog       |
+----------------+

14 modules enabled in total.

After one page load of that site's home page (/node).

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
9772 www-data  15   0  233m  33m  15m S  0.0  3.4   0:00.58 apache2
16324 mysql     15   0  230m  14m 3760 S  0.0  1.4  17:59.28 mysqld
9770 root      15   0  219m 9292 4540 S  0.0  0.9   0:00.05 apache2
9773 www-data  15   0  219m 6164 1336 S  0.0  0.6   0:00.01 apache2
9774 www-data  25   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9775 www-data  25   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9776 www-data  25   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9778 www-data  15   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9779 www-data  15   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
9780 www-data  15   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2 

Note that one apache process has increased in size, from 5MB to 33MB.

After several reloads of the home page:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
9772 www-data  16   0  233m  33m  15m S  0.0  3.4   0:00.76 apache2
9774 www-data  15   0  225m  20m  10m S  0.0  2.1   0:01.27 apache2
9775 www-data  15   0  225m  20m  10m S  0.0  2.1   0:01.30 apache2
9776 www-data  15   0  223m  18m  10m S  0.0  1.9   0:00.56 apache2
9778 www-data  15   0  223m  18m  10m S  0.0  1.8   0:00.34 apache2
9773 www-data  15   0  223m  17m  10m S  0.0  1.8   0:00.17 apache2
9780 www-data  16   0  223m  17m  10m S  0.0  1.8   0:00.16 apache2
9779 www-data  16   0  223m  17m  10m S  0.0  1.8   0:00.24 apache2
16324 mysql     15   0  230m  14m 3760 S  0.0  1.4  17:59.52 mysqld
9770 root      15   0  219m 9292 4540 S  0.0  0.9   0:00.05 apache2
9782 www-data  15   0  219m 6164 1336 S  0.0  0.6   0:00.00 apache2

So, the size of an apache process is 17 to 33 MB. Not optimal, but still acceptable.

On a server with 500MB, you can fit at least 15 Apache processes in memory.

We now conduct the same tests on another site, where the "open buffet binge syndrome" is evident:

This one has 113 modules enabled, as follows:

mysql> select name from system 
where type = 'module' and status = 1 
order by name;
+-------------------------------+
| name                          |
+-------------------------------+
| actions                       |
| activeselect                  |
| ad                            |
| adsense                       |
| adsense_injector              |
| advuser                       |
| ad_cache_file                 |
| ad_embed                      |
| ad_image                      |
| ad_notify                     |
| ad_report                     |
| ad_text                       |
| aggregator                    |
| akismet                       |
| basicevent                    |
| block                         |
| blog                          |
| calendar                      |
| calendar_ical                 |
| captcha                       |
| cck_field_perms               |
| color                         |
| content                       |
| content_taxonomy              |
| content_taxonomy_activeselect |
| content_taxonomy_autocomplete |
| content_taxonomy_options      |
| creativecommons_lite          |
| custom                        |
| custom_links                  |
| date                          |
| date_api                      |
| dblclick                      |
| devel                         |
| editview                      |
| email                         |
| event                         |
| eventrepeat                   |
| eventrepeat_views             |
| event_all_day                 |
| event_views                   |
| fckeditor                     |
| feedfield                     |
| fieldgroup                    |
| filter                        |
| fivestar                      |
| flag_content                  |
| form_collect                  |
| form_store                    |
| gmap                          |
| gmap_cck                      |
| gmap_location                 |
| gmap_macro_builder            |
| gmap_views                    |
| googleanalytics               |
| help                          |
| imagefield                    |
| insert_view                   |
| jscalendar                    |
| jstools                       |
| link                          |
| location                      |
| location_cookie               |
| location_views                |
| masquerade                    |
| menu                          |
| mimemail                      |
| node                          |
| nodecomment                   |
| nodefamily                    |
| nodequeue                     |
| nodereference                 |
| number                        |
| optionwidgets                 |
| pageroute                     |
| pageroute_nodefamily          |
| pageroute_ui                  |
| panels                        |
| path                          |
| pathauto                      |
| planet                        |
| profile                       |
| scheduler                     |
| search                        |
| send                          |
| statistics                    |
| subform_element               |
| system                        |
| taxonomy                      |
| taxonomySearch                |
| taxonomy_fields               |
| text                          |
| tinymce                       |
| tracker                       |
| update_status                 |
| upload                        |
| user                          |
| userreference                 |
| viewfield                     |
| views                         |
| views_argument_api            |
| views_bonus                   |
| views_bookmark                |
| views_fastsearch              |
| views_filterblock             |
| views_fusion                  |
| views_multiblock              |
| views_rss                     |
| views_theme_wizard            |
| views_ui                      |
| votingapi                     |
| watchdog                      |
| webform                       |
+-------------------------------+
113 rows in set (0.00 sec)

After a single page refresh to the home page, we see the following.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
9797 www-data  15   0  275m  93m  33m S  0.0  9.4   0:03.80 apache2
16324 mysql     15   0  230m  14m 3764 S  0.0  1.5  17:59.68 mysqld
9793 root      15   0  219m 9292 4540 S  0.0  0.9   0:00.05 apache2
9798 www-data  15   0  220m 6364 1332 S  0.0  0.6   0:00.02 apache2
9795 www-data  15   0  219m 6116 1296 S  0.0  0.6   0:00.00 apache2
9796 www-data  15   0  219m 6116 1296 S  0.0  0.6   0:00.00 apache2
9799 www-data  15   0  219m 5968 1148 S  0.0  0.6   0:00.00 apache2
10314 www-data  15   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2
10315 www-data  15   0  219m 5452  684 S  0.0  0.5   0:00.00 apache2

We see that the size of an Apache process is 93M (as opposed to 33M for a site with less modules).

After several page refreshes, things are as follows:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
9795 www-data  15   0  279m  98m  33m S  0.0  9.8   0:06.74 apache2
9799 www-data  15   0  279m  98m  33m S  0.0  9.8   0:06.52 apache2
9797 www-data  15   0  275m  93m  33m S  0.0  9.4   0:03.81 apache2
10314 www-data  15   0  275m  93m  33m S  0.0  9.4   0:03.73 apache2
10315 www-data  15   0  275m  93m  33m S  0.0  9.4   0:03.67 apache2
9798 www-data  15   0  275m  92m  34m S  0.0  9.2   0:04.24 apache2
16324 mysql     15   0  230m  15m 3880 S  0.0  1.5  18:00.42 mysqld
9793 root      15   0  219m 9292 4540 S  0.0  0.9   0:00.05 apache2
9796 www-data  15   0  220m 6440 1408 S  0.0  0.6   0:00.01 apache2 

The size of an apache process is 92 to 98 MB. That is way too much.

On a server with 500MB of available RAM, you can barely fit 5 processes at the same time in memory. Better have Apache's MaxClients tuned to 5 otherwise , if you get traffic spikes, your server will go to thrashing hell.

Don't even think of running such a site on shared hosting with medium traffic. Your host will quickly terminate your account for overuse of resources.

So far, full featured sites seem to have about 80 to 110 modules. I have seen a case where a site had 122 module.

Keeping the number of modules down can help a lot. See our article on whether a Drupal site can handle a million page views a day.

Update July 2010:

Check out our presentation on a site that does 2.8 million page views per day, 70 million per month, one server!.

That site broke the record later, and did 3.4 million page views in one day, and 92 million in one month.

Update September 2010:

We constantly see clients that go for an insanely large amount of modules. For example, we have seen several with 180 or so modules.

But it even goes beyond our worst nightmare. I once woke up from sleep with a nightmare that a client called us to help with a site that had 200 modules.

Then it was in real life that a client called about a site with 228 modules, and another had 231 modules! Later another one with 238 engaged us in consulting for that poor site.

Update January 2011

Here is a real life case of the effect of the number of modules on importing of new nodes. David Kent Norman has an article detailing how he made things much more speedy by disabling modules.

Update October 2012

New world record for number of enabled modules: 381 modules! More details at Presentation: Huge! Drupal site with 381 modules, 174 GB MySQL database and tables over 200 million rows. The good news is that we were able to get that down to 207 modules only. It can be done!