In a previous article, I described the "open buffet binge syndrome" where sites overconsume contrib modules, and gave examples where Apache memory was more than 100MB per process.

Today, I tried to take that further, and trace where this memory is consumed and by which modules. This lead to several interesting tangents.

I created a patch for Drupal 6 that can trace this information, and ran it on a replica of a live site with too many modules running on Drupal 5.1, to a default install of Drupal 6 HEAD (checked out Aug 27).

I then compared a plain default install of Drupal 6.x site to a plain Drupal 5.2 site, and then separated the amount of memory used by modules from the memory used by Drupal's bootstrap (inferred).

Note that this measurement is not the entire story, because depending on the underlying data and the exact page visited, memory consumption would be more due to loading/populating data structures.

Here is the table of the data, comparing a bloated Drupal 5.1 site to Drupal 6.x (Aug 27, 2007).

Update: I attached the results in Open Office and Excel format. There are two sheets in there. 

Drupal 5.1 bloated site vs. Drupal 6

Memory usage comparison            
             
Drupal 5.1 site with lots of modules       Drupal 6.x site (27 Aug 2007) 27/08/07  
  Bytes Kilobytes     Bytes Kilobytes
Start memory 111448 108.84   Start memory 109848 107.27
End memory 51984944 50766.55   End memory 11623920 11351.48
Total used by Drupal 51873496 50657.71   Total used by Drupal 11514072 11244.21
Total used by modules 40898088 39939.54   Total used by modules 6194104 6048.93
Total used by Drupal bootstrap 10975408 10718.17   Total used by Drupal bootstrap 5319968 5195.28
             
location 1859024 1815.45   user 1592112 1554.8
eventrepeat 1726008 1685.55   node 953576 931.23
event 1583752 1546.63   comment 908536 887.24
taxonomy_fields 1577728 1540.75   filter 702496 686.03
views_filterblock 1562264 1525.65   system 509360 497.42
user 1489456 1454.55   taxonomy 439080 428.79
node 1371624 1339.48   menu 405408 395.91
ad 1282920 1252.85   color 280968 274.38
system 1077256 1052.01   block 209832 204.91
date 1005552 981.98   update 128120 125.12
views_ui 960832 938.31   dblog 48488 47.35
calendar 924168 902.51   help 16128 15.75
views 906232 884.99        
webform 744608 727.16        
aggregator 728008 710.95        
pathauto 711272 694.60        
filter 697480 681.13        
taxonomy 674888 659.07        
devel 665528 649.93        
planet 638648 623.68        
nodequeue 630488 615.71        
tinymce 558088 545.01        
adsense 491744 480.22        
update_status 485256 473.88        
nodecomment 476560 465.39        
search 469648 458.64        
content 464568 453.68        
views_bookmark 457552 446.83        
location_views 453064 442.45        
pageroute 446184 435.73        
views_theme_wizard 444472 434.05        
profile 439904 429.59        
nodefamily 430456 420.37        
upload 425376 415.41        
actions 413752 404.05        
gmap 412168 402.51        
advuser 406936 397.40        
panels 371912 363.20        
ad_notify 355280 346.95        
block 338896 330.95        
menu 336360 328.48        
views_bonus 326616 318.96        
pageroute_ui 311672 304.37        
gmap_location 300104 293.07        
statistics 299856 292.83        
imagefield 286256 279.55        
color 274736 268.30        
fieldgroup 269288 262.98        
fivestar 261416 255.29        
flag_content 234400 228.91        
ad_image 220736 215.56        
event_views 219872 214.72        
feedfield 219032 213.90        
link 211344 206.39        
nodereference 202848 198.09        
calendar_ical 201872 197.14        
votingapi 200352 195.66        
masquerade 195496 190.91        
ad_embed 183712 179.41        
eventrepeat_views 180072 175.85        
views_fusion 174336 170.25        
editview 171880 167.85        
number 162576 158.77        
views_multiblock 161440 157.66        
path 158224 154.52        
email 157192 153.51        
custom_links 151440 147.89        
text 150696 147.16        
blog 142888 139.54        
views_argument_api 137520 134.30        
send 135240 132.07        
creativecommons_lite 131352 128.27        
gmap_cck 128784 125.77        
cck_field_perms 124632 121.71        
content_taxonomy_activeselect 120792 117.96        
form_store 120328 117.51        
mimemail 119920 117.11        
ad_text 119056 116.27        
watchdog 117616 114.86        
captcha 113184 110.53        
scheduler 110000 107.42        
content_taxonomy_autocomplete 109856 107.28        
ad_cache_file 109608 107.04        
userreference 106936 104.43        
pageroute_nodefamily 101952 99.56        
content_taxonomy 93648 91.45        
ad_report 91832 89.68        
adsense_injector 84896 82.91        
viewfield 84608 82.63        
content_taxonomy_options 83400 81.45        
views_rss 80696 78.80        
gmap_views 79016 77.16        
subform_element 78616 76.77        
views_fastsearch 75704 73.93        
optionwidgets 74200 72.46        
googleanalytics 73832 72.10        
tracker 68368 66.77        
taxonomySearch 63200 61.72        
jscalendar 56280 54.96        
gmap_macro_builder 54240 52.97        
help 53400 52.15        
dblclick 48920 47.77        
insert_view 42016 41.03        
activeselect 41528 40.55        
location_cookie 33080 32.30        
jstools 26712 26.09        
basicevent 26096 25.48        
form_collect 20456 19.98        
custom 14528 14.19        
event_all_day 11568 11.30        
date_api 4208 4.11        

As you can see, the number of modules can make the difference between using 11MB and 50MB!

Comparison of basic Drupal 5.2 and Drupal 6.x

Drupal 6.x site (27 Aug) 27/08/07     Drupal 5.2, basic install      
  Bytes Kilobytes     Bytes Kilobytes Difference
Start memory 110000 107.42   Start memory 109672 107.10 -328
End memory 11624032 11351.59   End memory 11662272 11388.94 38240
Total used by Drupal 11514032 11244.17   Total used by Drupal 11552600 11281.84 38568
Total used by modules 6194104 6048.93   Total used by modules 7382400 7209.38 1188296
Total used by bootstrap 5319928 5195.24   Total used by bootstrap 4170200 4072.46 -1149728
               
user 1592112 1554.8   user 1493544 1458.54  
node 953568 931.22   node 1396216 1363.49  
comment 908504 887.21   system 1080176 1054.86  
filter 702472 686.01   comment 876928 856.38  
system 509352 497.41   taxonomy 709824 693.19  
taxonomy 439056 428.77   filter 694800 678.52  
menu 405408 395.91   menu 344544 336.47  
color 280968 274.38   block 338704 330.77  
block 209832 204.91   color 276784 270.30  
update 128120 125.12   watchdog 117752 114.99  
dblog 48584 47.45   help 53128 51.88  
help 16128 15.75          

I have not seen any significant difference between having APC on and having it disabled.

The patch for Drupal 6 and Drupal 5.2 are attached for those who want to use them.

Where does this go from here? Several things I like to see happen:

  • The results of these tests need to be replicated by a few other people.
  • The differences between Drupal 5.2 and Drupal 6 need to analyzed and see whether they warrant more changes. The split mode did improve the modules part, but the Drupal bootstrap part has worstened a bit.
  • The includes/module.inc part to go into core, so measurement can be done without core modifications.
  • The parts in index.php should move to the devel module, so measurement can be done by that module.

Thoughts? Comments?

Update 2008 October 22: I attached two new completely rearchitected patches, one for Drupal 5.12 and one for Drupal 6.6, both came out today.

The patch will not do anything until you add this configuration variable to your settings.php file:

$conf['memory_profiling'] = 1;

The patch now appends the results to a file called memory.csv in Drupal's temp directory, e.g. /tmp, or what you have set it to if different from the default.

The data in the file is not sorted, but can easily be loaded in OpenOffice or Excel and reformatted in any way you like.

Update 2009-09-30:

The re-architected patch has been rerolled for Drupal 6.14, the latest version for now.

Any help in porting this to Drupal 7.x, with the new registery is appreciated.

Update 2010-07-04:

Please note that the devel project now contains the "performance" module written by the author of this article. It provides a much more accurate measurement than the above patches, and require no patching of core. It also allows measurement of different URLs on the site.

Update 2012-09-19:

The patch has been ported to Drupal 7.x by Anton, and is attached to this article.

Comments

Tue, 2007/08/28 - 11:15

Hi, nice reports! In my browser page aren't displayed correctly.. can you provide a PDF version or something that don't breaks line-returns?

I'd like to test your patches on my next-on-production website.. I'll let you know the results!
Bye!

Tue, 2007/08/28 - 15:51

Hi,

Thanks for planning to test this.

I attached the report in Open Office and Excel format. There are two sheets in that file, one for Drupal 5.1 (open buffet) vs. Drupal 6, and the other for a Drupal 5.2 minimal install vs. Drupal 6.

Let us know the results.
--
2bits -- Drupal and Backdrop CMS consulting

Thu, 2007/09/27 - 12:35

Thanks for the drupal-5 patch, it made clear to me that some modules really aren't needed on a live site.

Like update_status and image_import.

Mon, 2011/10/10 - 01:55

I'm not an expert in Drupal.But I think it do not take 3500mb.I also have around 35mb per page also, and thats with an opcode cache and memcache.I'd really like this discussion to go further.

Mon, 2008/11/03 - 11:36

I just ran your 5.12 patch and notice the permissions were not set properly? It placed, on a windows system, everyone and none on the patched files when they should be set to System and admin under security. Is this by design or am I to set the permissions as I patch?

http://www.nisekonews.com is a small Drupal 5.2 / G2 site with a few modules on shared hosting with 32M of memory. Every few weeks or so the site will stop working and only serve 500 errors. The host http://www.ezoshosting.com has found that they must increase the memory to 512M and then the error will go away. Once the error goes away they can decrease the memory back to 32M.

The sites current Drupal module memory footprint can be found at the footer of the site at
http://www.niskeonews.com (unless I need to remove it by the time you read this). The memory use today is posted below as well as some server info.

The server is RedHat, Apache 1.3.37, PHP 4.4.7, and MySQL 5.0.27-standard.
Drupal is drupal-5.2.tar.gz, brushed_steel-5.x-1.2.tar.gz, g2image-v3.0-RC1.zip, gallery-2.2.2-full.tar.gz, gallery-5.x-2.x-dev.tar.gz,
article-5.x-2.2.tar.gz, directory-5.x-2.0.tar.gz, google_analytics-5.x-1.x-dev.tar.gz, lightbox2-5.x-1.x-dev.tar.gz, lightbox2.03.3.zip, panels-5.x-1.2.tar.gz, scriptaculous-drupal.zip,
tinymce-5.x.tar.gz, tinymce_2_1_1_1.tgz, tinymce_compressor_php_1_1_0.tgz, views-5.x-1.6.tar.gz,
weather-3.x-3.1.tar.gz

The memory for today is below. I will be checking over time to see if there is a memory leak and where ...
Today,September 27, 2007, 8:30 am
Start,62784
End,29952112
Total,29889328
user,1195944
node,1143672
system,882304
views,814504
views_ui,808736
comment,702808
taxonomy,599240
filter,579432
weather,511392
tinymce,482080
forum,442168
directory,360624
upload,354640
panels,327368
gallery,311552
block,297248
menu,279088
poll,238240
statistics,226728
contact,215376
article,149968
blog,113848
lightbox2,97752
googleanalytics,94072
watchdog,92384
gallery_menu,78536
Modules,11399704
Bootstrap,18489624

Thu, 2007/10/11 - 04:30

I swapped the statistics module for the google analytic module and disabled the watchdog query and removed the watch dog module as well as employed the low memory patch from
http://wtanaka.com/drupal/system

My memory stats are now lower
Today,October 11, 2007, 12:20 pm
Start,63280
End,28220072
Total,28156792
user,1195128
node,1143672
views,814504
views_ui,808736
comment,702808
taxonomy,599264
filter,579432
weather,509032
tinymce,482080
forum,442168
directory,360624
panels,327368
gallery,311632
block,297360
menu,279088
poll,238240
contact,215376
system,165752
article,149912
blog,113848
lightbox2,97752
googleanalytics,94368
gallery_menu,78536
Modules,10006680
Bootstrap,18150112

Thanks for your help. If the problem continues I will investigate further but am disabling the module memory reporting for now.

Mon, 2007/10/22 - 01:01

Well the Ad module was turned on the 500 errors started popping up intermittenly ... I grabbed this memory foot print and log before disabling the Ad module ...

Apache error log ...
Its only three three files, mostly the index file, sometime the ad file, and maybe once the cron file ... cron set to run once per day
Cannot allocate memory: couldn't spawn child process: /drupal/index.php
Cannot allocate memory: couldn't spawn child process: /drupal/modules/ad/adserve.php
Cannot allocate memory: couldn't spawn child process: /drupal/cron.php

The memory was ...
Total,29500064
ad,1075000

After the ad module was off ...
Total,28662592

Wed, 2008/05/07 - 02:38

This is just what I needed!

I am on hostmonster (awardspace before that), and they give an error page each time I go over 32M.

Turns out all of the modules consume quit a bit of memory! Simplenews goes over 2M! Just for a newsletter subscription!

Does anybody know of a good host? I mean really good for a Drupal site? I am thinking nearly dedicated, but still a bit affordable.

I am considering 2mhost, they offer 64M.

Another question. If I host multiple site on a domain. How will that affect the memory or won't it?

Pages

Is your Drupal or Backdrop CMS site slow?
Is it suffering from server resources shortages?
Is it experiencing outages?
Contact us for Drupal or Backdrop CMS Performance Optimization and Tuning Consulting