In a previous article, I described the "open buffet binge syndrome" where sites overconsume contrib modules, and gave examples where Apache memory was more than 100MB per process.
Today, I tried to take that further, and trace where this memory is consumed and by which modules. This lead to several interesting tangents.
I created a patch for Drupal 6 that can trace this information, and ran it on a replica of a live site with too many modules running on Drupal 5.1, to a default install of Drupal 6 HEAD (checked out Aug 27).
I then compared a plain default install of Drupal 6.x site to a plain Drupal 5.2 site, and then separated the amount of memory used by modules from the memory used by Drupal's bootstrap (inferred).
Note that this measurement is not the entire story, because depending on the underlying data and the exact page visited, memory consumption would be more due to loading/populating data structures.
Here is the table of the data, comparing a bloated Drupal 5.1 site to Drupal 6.x (Aug 27, 2007).
Update: I attached the results in Open Office and Excel format. There are two sheets in there.
Drupal 5.1 bloated site vs. Drupal 6
Memory usage comparison | ||||||
Drupal 5.1 site with lots of modules | Drupal 6.x site (27 Aug 2007) | 27/08/07 | ||||
Bytes | Kilobytes | Bytes | Kilobytes | |||
Start memory | 111448 | 108.84 | Start memory | 109848 | 107.27 | |
End memory | 51984944 | 50766.55 | End memory | 11623920 | 11351.48 | |
Total used by Drupal | 51873496 | 50657.71 | Total used by Drupal | 11514072 | 11244.21 | |
Total used by modules | 40898088 | 39939.54 | Total used by modules | 6194104 | 6048.93 | |
Total used by Drupal bootstrap | 10975408 | 10718.17 | Total used by Drupal bootstrap | 5319968 | 5195.28 | |
location | 1859024 | 1815.45 | user | 1592112 | 1554.8 | |
eventrepeat | 1726008 | 1685.55 | node | 953576 | 931.23 | |
event | 1583752 | 1546.63 | comment | 908536 | 887.24 | |
taxonomy_fields | 1577728 | 1540.75 | filter | 702496 | 686.03 | |
views_filterblock | 1562264 | 1525.65 | system | 509360 | 497.42 | |
user | 1489456 | 1454.55 | taxonomy | 439080 | 428.79 | |
node | 1371624 | 1339.48 | menu | 405408 | 395.91 | |
ad | 1282920 | 1252.85 | color | 280968 | 274.38 | |
system | 1077256 | 1052.01 | block | 209832 | 204.91 | |
date | 1005552 | 981.98 | update | 128120 | 125.12 | |
views_ui | 960832 | 938.31 | dblog | 48488 | 47.35 | |
calendar | 924168 | 902.51 | help | 16128 | 15.75 | |
views | 906232 | 884.99 | ||||
webform | 744608 | 727.16 | ||||
aggregator | 728008 | 710.95 | ||||
pathauto | 711272 | 694.60 | ||||
filter | 697480 | 681.13 | ||||
taxonomy | 674888 | 659.07 | ||||
devel | 665528 | 649.93 | ||||
planet | 638648 | 623.68 | ||||
nodequeue | 630488 | 615.71 | ||||
tinymce | 558088 | 545.01 | ||||
adsense | 491744 | 480.22 | ||||
update_status | 485256 | 473.88 | ||||
nodecomment | 476560 | 465.39 | ||||
search | 469648 | 458.64 | ||||
content | 464568 | 453.68 | ||||
views_bookmark | 457552 | 446.83 | ||||
location_views | 453064 | 442.45 | ||||
pageroute | 446184 | 435.73 | ||||
views_theme_wizard | 444472 | 434.05 | ||||
profile | 439904 | 429.59 | ||||
nodefamily | 430456 | 420.37 | ||||
upload | 425376 | 415.41 | ||||
actions | 413752 | 404.05 | ||||
gmap | 412168 | 402.51 | ||||
advuser | 406936 | 397.40 | ||||
panels | 371912 | 363.20 | ||||
ad_notify | 355280 | 346.95 | ||||
block | 338896 | 330.95 | ||||
menu | 336360 | 328.48 | ||||
views_bonus | 326616 | 318.96 | ||||
pageroute_ui | 311672 | 304.37 | ||||
gmap_location | 300104 | 293.07 | ||||
statistics | 299856 | 292.83 | ||||
imagefield | 286256 | 279.55 | ||||
color | 274736 | 268.30 | ||||
fieldgroup | 269288 | 262.98 | ||||
fivestar | 261416 | 255.29 | ||||
flag_content | 234400 | 228.91 | ||||
ad_image | 220736 | 215.56 | ||||
event_views | 219872 | 214.72 | ||||
feedfield | 219032 | 213.90 | ||||
link | 211344 | 206.39 | ||||
nodereference | 202848 | 198.09 | ||||
calendar_ical | 201872 | 197.14 | ||||
votingapi | 200352 | 195.66 | ||||
masquerade | 195496 | 190.91 | ||||
ad_embed | 183712 | 179.41 | ||||
eventrepeat_views | 180072 | 175.85 | ||||
views_fusion | 174336 | 170.25 | ||||
editview | 171880 | 167.85 | ||||
number | 162576 | 158.77 | ||||
views_multiblock | 161440 | 157.66 | ||||
path | 158224 | 154.52 | ||||
157192 | 153.51 | |||||
custom_links | 151440 | 147.89 | ||||
text | 150696 | 147.16 | ||||
blog | 142888 | 139.54 | ||||
views_argument_api | 137520 | 134.30 | ||||
send | 135240 | 132.07 | ||||
creativecommons_lite | 131352 | 128.27 | ||||
gmap_cck | 128784 | 125.77 | ||||
cck_field_perms | 124632 | 121.71 | ||||
content_taxonomy_activeselect | 120792 | 117.96 | ||||
form_store | 120328 | 117.51 | ||||
mimemail | 119920 | 117.11 | ||||
ad_text | 119056 | 116.27 | ||||
watchdog | 117616 | 114.86 | ||||
captcha | 113184 | 110.53 | ||||
scheduler | 110000 | 107.42 | ||||
content_taxonomy_autocomplete | 109856 | 107.28 | ||||
ad_cache_file | 109608 | 107.04 | ||||
userreference | 106936 | 104.43 | ||||
pageroute_nodefamily | 101952 | 99.56 | ||||
content_taxonomy | 93648 | 91.45 | ||||
ad_report | 91832 | 89.68 | ||||
adsense_injector | 84896 | 82.91 | ||||
viewfield | 84608 | 82.63 | ||||
content_taxonomy_options | 83400 | 81.45 | ||||
views_rss | 80696 | 78.80 | ||||
gmap_views | 79016 | 77.16 | ||||
subform_element | 78616 | 76.77 | ||||
views_fastsearch | 75704 | 73.93 | ||||
optionwidgets | 74200 | 72.46 | ||||
googleanalytics | 73832 | 72.10 | ||||
tracker | 68368 | 66.77 | ||||
taxonomySearch | 63200 | 61.72 | ||||
jscalendar | 56280 | 54.96 | ||||
gmap_macro_builder | 54240 | 52.97 | ||||
help | 53400 | 52.15 | ||||
dblclick | 48920 | 47.77 | ||||
insert_view | 42016 | 41.03 | ||||
activeselect | 41528 | 40.55 | ||||
location_cookie | 33080 | 32.30 | ||||
jstools | 26712 | 26.09 | ||||
basicevent | 26096 | 25.48 | ||||
form_collect | 20456 | 19.98 | ||||
custom | 14528 | 14.19 | ||||
event_all_day | 11568 | 11.30 | ||||
date_api | 4208 | 4.11 |
As you can see, the number of modules can make the difference between using 11MB and 50MB!
Comparison of basic Drupal 5.2 and Drupal 6.x
Drupal 6.x site (27 Aug) | 27/08/07 | Drupal 5.2, basic install | |||||
Bytes | Kilobytes | Bytes | Kilobytes | Difference | |||
Start memory | 110000 | 107.42 | Start memory | 109672 | 107.10 | -328 | |
End memory | 11624032 | 11351.59 | End memory | 11662272 | 11388.94 | 38240 | |
Total used by Drupal | 11514032 | 11244.17 | Total used by Drupal | 11552600 | 11281.84 | 38568 | |
Total used by modules | 6194104 | 6048.93 | Total used by modules | 7382400 | 7209.38 | 1188296 | |
Total used by bootstrap | 5319928 | 5195.24 | Total used by bootstrap | 4170200 | 4072.46 | -1149728 | |
user | 1592112 | 1554.8 | user | 1493544 | 1458.54 | ||
node | 953568 | 931.22 | node | 1396216 | 1363.49 | ||
comment | 908504 | 887.21 | system | 1080176 | 1054.86 | ||
filter | 702472 | 686.01 | comment | 876928 | 856.38 | ||
system | 509352 | 497.41 | taxonomy | 709824 | 693.19 | ||
taxonomy | 439056 | 428.77 | filter | 694800 | 678.52 | ||
menu | 405408 | 395.91 | menu | 344544 | 336.47 | ||
color | 280968 | 274.38 | block | 338704 | 330.77 | ||
block | 209832 | 204.91 | color | 276784 | 270.30 | ||
update | 128120 | 125.12 | watchdog | 117752 | 114.99 | ||
dblog | 48584 | 47.45 | help | 53128 | 51.88 | ||
help | 16128 | 15.75 |
I have not seen any significant difference between having APC on and having it disabled.
The patch for Drupal 6 and Drupal 5.2 are attached for those who want to use them.
Where does this go from here? Several things I like to see happen:
- The results of these tests need to be replicated by a few other people.
- The differences between Drupal 5.2 and Drupal 6 need to analyzed and see whether they warrant more changes. The split mode did improve the modules part, but the Drupal bootstrap part has worstened a bit.
- The includes/module.inc part to go into core, so measurement can be done without core modifications.
- The parts in index.php should move to the devel module, so measurement can be done by that module.
Thoughts? Comments?
Update 2008 October 22: I attached two new completely rearchitected patches, one for Drupal 5.12 and one for Drupal 6.6, both came out today.
The patch will not do anything until you add this configuration variable to your settings.php file:
$conf['memory_profiling'] = 1;
The patch now appends the results to a file called memory.csv in Drupal's temp directory, e.g. /tmp, or what you have set it to if different from the default.
The data in the file is not sorted, but can easily be loaded in OpenOffice or Excel and reformatted in any way you like.
Update 2009-09-30:
The re-architected patch has been rerolled for Drupal 6.14, the latest version for now.
Any help in porting this to Drupal 7.x, with the new registery is appreciated.
Update 2010-07-04:
Please note that the devel project now contains the "performance" module written by the author of this article. It provides a much more accurate measurement than the above patches, and require no patching of core. It also allows measurement of different URLs on the site.
Update 2012-09-19:
The patch has been ported to Drupal 7.x by Anton, and is attached to this article.
Attachment | Size |
---|---|
Drupal 6x. memory (Old).patch | 1.62 KB |
Drupal 5.2 memory (Old).patch | 1.68 KB |
drupal-memory.ods | 21.54 KB |
drupal-memory.xls | 116 KB |
Drupal 6.6 memory.patch | 7.82 KB |
Drupal 5.12 memory.patch | 7.26 KB |
drupal-memory-6.14.patch | 5.38 KB |
Comments
Visitor (not verified)
Have a look at the drupal 6.6
Sun, 2009/11/29 - 17:49Have a look at the drupal 6.6 patch, the code is in there.
But it would definitely be nice to see it in the latest patch.
Visitor (not verified)
it rocks
Wed, 2009/12/02 - 05:59Thank you very much for your patch
I was searching too for the memory.inc and found it in drupal-memory-6.6.patch file
I use drupal 6.14
Your patch gives me eyes to see that drupal was called for some images (png), i will change my htaccess right now
Thanks Again
Visitor (not verified)
Help installing on drupal 6.13
Mon, 2010/03/01 - 12:11I'd really appreciate it if someone could help me install this patch. I've created includes/memory.inc from the ver 6.6 patch, installed 6.14 ver patch and added config line to settings.php, however the module refuses to give any output to my temp directory. Any clues?
Visitor (not verified)
Thanks for the great patch. I
Wed, 2010/03/24 - 11:04Thanks for the great patch. I ran into a problem, however, where the 6.14 patch doesn't include the memory.inc part (i got it from the 6.6 version). Otherwise perfect :)
Visitor (not verified)
Update for Drupal 6.16
Thu, 2010/03/25 - 10:30Could someone please update the patch to work with Drupal 6.16?
I tried it and theme.inc keeps getting rejected, they changed a lot in there. :(
Visitor (not verified)
Update for patched Drupal 6,16
Mon, 2010/04/12 - 08:23maybe what we would really need is a patch for (D6.16 + patched includes/bootstrap.inc) as per http://drupal.org/node/732064#comment-2788744
Khalid
performance.module in devel project
Mon, 2010/04/12 - 09:52There is already a way: install the devel module, and under it there is a performance module that does several things, including measure memory consumption for each page.
Visitor (not verified)
Files changed to drupal version 6.17
Sun, 2010/07/04 - 11:51Hi There
First of all great work, It helped me a lot.
I would like to contribute a bit by sending you to the following link . This should do the work for Drupal 6.17. I am sorry I dod not attached the patch file as I am a bit short in time and I have no familiarity with the patches stuff.
Visitor (not verified)
Pressflow 6.19
Wed, 2010/12/08 - 08:22I've added a version of this patch for Pressflow 6.19. I don't know if it works with vanilla Drupal or not--it may need to be massaged a bit.
http://drupal.org/node/844176#comment-3802302
Dorien (not verified)
Drupal 7
Sun, 2011/05/29 - 12:48Any idea on how to modify this patch for Drupal 7?
Pages