In a previous article, I described the "open buffet binge syndrome" where sites overconsume contrib modules, and gave examples where Apache memory was more than 100MB per process.
Today, I tried to take that further, and trace where this memory is consumed and by which modules. This lead to several interesting tangents.
I created a patch for Drupal 6 that can trace this information, and ran it on a replica of a live site with too many modules running on Drupal 5.1, to a default install of Drupal 6 HEAD (checked out Aug 27).
I then compared a plain default install of Drupal 6.x site to a plain Drupal 5.2 site, and then separated the amount of memory used by modules from the memory used by Drupal's bootstrap (inferred).
Note that this measurement is not the entire story, because depending on the underlying data and the exact page visited, memory consumption would be more due to loading/populating data structures.
Here is the table of the data, comparing a bloated Drupal 5.1 site to Drupal 6.x (Aug 27, 2007).
Update: I attached the results in Open Office and Excel format. There are two sheets in there.
Drupal 5.1 bloated site vs. Drupal 6
Memory usage comparison | ||||||
Drupal 5.1 site with lots of modules | Drupal 6.x site (27 Aug 2007) | 27/08/07 | ||||
Bytes | Kilobytes | Bytes | Kilobytes | |||
Start memory | 111448 | 108.84 | Start memory | 109848 | 107.27 | |
End memory | 51984944 | 50766.55 | End memory | 11623920 | 11351.48 | |
Total used by Drupal | 51873496 | 50657.71 | Total used by Drupal | 11514072 | 11244.21 | |
Total used by modules | 40898088 | 39939.54 | Total used by modules | 6194104 | 6048.93 | |
Total used by Drupal bootstrap | 10975408 | 10718.17 | Total used by Drupal bootstrap | 5319968 | 5195.28 | |
location | 1859024 | 1815.45 | user | 1592112 | 1554.8 | |
eventrepeat | 1726008 | 1685.55 | node | 953576 | 931.23 | |
event | 1583752 | 1546.63 | comment | 908536 | 887.24 | |
taxonomy_fields | 1577728 | 1540.75 | filter | 702496 | 686.03 | |
views_filterblock | 1562264 | 1525.65 | system | 509360 | 497.42 | |
user | 1489456 | 1454.55 | taxonomy | 439080 | 428.79 | |
node | 1371624 | 1339.48 | menu | 405408 | 395.91 | |
ad | 1282920 | 1252.85 | color | 280968 | 274.38 | |
system | 1077256 | 1052.01 | block | 209832 | 204.91 | |
date | 1005552 | 981.98 | update | 128120 | 125.12 | |
views_ui | 960832 | 938.31 | dblog | 48488 | 47.35 | |
calendar | 924168 | 902.51 | help | 16128 | 15.75 | |
views | 906232 | 884.99 | ||||
webform | 744608 | 727.16 | ||||
aggregator | 728008 | 710.95 | ||||
pathauto | 711272 | 694.60 | ||||
filter | 697480 | 681.13 | ||||
taxonomy | 674888 | 659.07 | ||||
devel | 665528 | 649.93 | ||||
planet | 638648 | 623.68 | ||||
nodequeue | 630488 | 615.71 | ||||
tinymce | 558088 | 545.01 | ||||
adsense | 491744 | 480.22 | ||||
update_status | 485256 | 473.88 | ||||
nodecomment | 476560 | 465.39 | ||||
search | 469648 | 458.64 | ||||
content | 464568 | 453.68 | ||||
views_bookmark | 457552 | 446.83 | ||||
location_views | 453064 | 442.45 | ||||
pageroute | 446184 | 435.73 | ||||
views_theme_wizard | 444472 | 434.05 | ||||
profile | 439904 | 429.59 | ||||
nodefamily | 430456 | 420.37 | ||||
upload | 425376 | 415.41 | ||||
actions | 413752 | 404.05 | ||||
gmap | 412168 | 402.51 | ||||
advuser | 406936 | 397.40 | ||||
panels | 371912 | 363.20 | ||||
ad_notify | 355280 | 346.95 | ||||
block | 338896 | 330.95 | ||||
menu | 336360 | 328.48 | ||||
views_bonus | 326616 | 318.96 | ||||
pageroute_ui | 311672 | 304.37 | ||||
gmap_location | 300104 | 293.07 | ||||
statistics | 299856 | 292.83 | ||||
imagefield | 286256 | 279.55 | ||||
color | 274736 | 268.30 | ||||
fieldgroup | 269288 | 262.98 | ||||
fivestar | 261416 | 255.29 | ||||
flag_content | 234400 | 228.91 | ||||
ad_image | 220736 | 215.56 | ||||
event_views | 219872 | 214.72 | ||||
feedfield | 219032 | 213.90 | ||||
link | 211344 | 206.39 | ||||
nodereference | 202848 | 198.09 | ||||
calendar_ical | 201872 | 197.14 | ||||
votingapi | 200352 | 195.66 | ||||
masquerade | 195496 | 190.91 | ||||
ad_embed | 183712 | 179.41 | ||||
eventrepeat_views | 180072 | 175.85 | ||||
views_fusion | 174336 | 170.25 | ||||
editview | 171880 | 167.85 | ||||
number | 162576 | 158.77 | ||||
views_multiblock | 161440 | 157.66 | ||||
path | 158224 | 154.52 | ||||
157192 | 153.51 | |||||
custom_links | 151440 | 147.89 | ||||
text | 150696 | 147.16 | ||||
blog | 142888 | 139.54 | ||||
views_argument_api | 137520 | 134.30 | ||||
send | 135240 | 132.07 | ||||
creativecommons_lite | 131352 | 128.27 | ||||
gmap_cck | 128784 | 125.77 | ||||
cck_field_perms | 124632 | 121.71 | ||||
content_taxonomy_activeselect | 120792 | 117.96 | ||||
form_store | 120328 | 117.51 | ||||
mimemail | 119920 | 117.11 | ||||
ad_text | 119056 | 116.27 | ||||
watchdog | 117616 | 114.86 | ||||
captcha | 113184 | 110.53 | ||||
scheduler | 110000 | 107.42 | ||||
content_taxonomy_autocomplete | 109856 | 107.28 | ||||
ad_cache_file | 109608 | 107.04 | ||||
userreference | 106936 | 104.43 | ||||
pageroute_nodefamily | 101952 | 99.56 | ||||
content_taxonomy | 93648 | 91.45 | ||||
ad_report | 91832 | 89.68 | ||||
adsense_injector | 84896 | 82.91 | ||||
viewfield | 84608 | 82.63 | ||||
content_taxonomy_options | 83400 | 81.45 | ||||
views_rss | 80696 | 78.80 | ||||
gmap_views | 79016 | 77.16 | ||||
subform_element | 78616 | 76.77 | ||||
views_fastsearch | 75704 | 73.93 | ||||
optionwidgets | 74200 | 72.46 | ||||
googleanalytics | 73832 | 72.10 | ||||
tracker | 68368 | 66.77 | ||||
taxonomySearch | 63200 | 61.72 | ||||
jscalendar | 56280 | 54.96 | ||||
gmap_macro_builder | 54240 | 52.97 | ||||
help | 53400 | 52.15 | ||||
dblclick | 48920 | 47.77 | ||||
insert_view | 42016 | 41.03 | ||||
activeselect | 41528 | 40.55 | ||||
location_cookie | 33080 | 32.30 | ||||
jstools | 26712 | 26.09 | ||||
basicevent | 26096 | 25.48 | ||||
form_collect | 20456 | 19.98 | ||||
custom | 14528 | 14.19 | ||||
event_all_day | 11568 | 11.30 | ||||
date_api | 4208 | 4.11 |
As you can see, the number of modules can make the difference between using 11MB and 50MB!
Comparison of basic Drupal 5.2 and Drupal 6.x
Drupal 6.x site (27 Aug) | 27/08/07 | Drupal 5.2, basic install | |||||
Bytes | Kilobytes | Bytes | Kilobytes | Difference | |||
Start memory | 110000 | 107.42 | Start memory | 109672 | 107.10 | -328 | |
End memory | 11624032 | 11351.59 | End memory | 11662272 | 11388.94 | 38240 | |
Total used by Drupal | 11514032 | 11244.17 | Total used by Drupal | 11552600 | 11281.84 | 38568 | |
Total used by modules | 6194104 | 6048.93 | Total used by modules | 7382400 | 7209.38 | 1188296 | |
Total used by bootstrap | 5319928 | 5195.24 | Total used by bootstrap | 4170200 | 4072.46 | -1149728 | |
user | 1592112 | 1554.8 | user | 1493544 | 1458.54 | ||
node | 953568 | 931.22 | node | 1396216 | 1363.49 | ||
comment | 908504 | 887.21 | system | 1080176 | 1054.86 | ||
filter | 702472 | 686.01 | comment | 876928 | 856.38 | ||
system | 509352 | 497.41 | taxonomy | 709824 | 693.19 | ||
taxonomy | 439056 | 428.77 | filter | 694800 | 678.52 | ||
menu | 405408 | 395.91 | menu | 344544 | 336.47 | ||
color | 280968 | 274.38 | block | 338704 | 330.77 | ||
block | 209832 | 204.91 | color | 276784 | 270.30 | ||
update | 128120 | 125.12 | watchdog | 117752 | 114.99 | ||
dblog | 48584 | 47.45 | help | 53128 | 51.88 | ||
help | 16128 | 15.75 |
I have not seen any significant difference between having APC on and having it disabled.
The patch for Drupal 6 and Drupal 5.2 are attached for those who want to use them.
Where does this go from here? Several things I like to see happen:
- The results of these tests need to be replicated by a few other people.
- The differences between Drupal 5.2 and Drupal 6 need to analyzed and see whether they warrant more changes. The split mode did improve the modules part, but the Drupal bootstrap part has worstened a bit.
- The includes/module.inc part to go into core, so measurement can be done without core modifications.
- The parts in index.php should move to the devel module, so measurement can be done by that module.
Thoughts? Comments?
Update 2008 October 22: I attached two new completely rearchitected patches, one for Drupal 5.12 and one for Drupal 6.6, both came out today.
The patch will not do anything until you add this configuration variable to your settings.php file:
$conf['memory_profiling'] = 1;
The patch now appends the results to a file called memory.csv in Drupal's temp directory, e.g. /tmp, or what you have set it to if different from the default.
The data in the file is not sorted, but can easily be loaded in OpenOffice or Excel and reformatted in any way you like.
Update 2009-09-30:
The re-architected patch has been rerolled for Drupal 6.14, the latest version for now.
Any help in porting this to Drupal 7.x, with the new registery is appreciated.
Update 2010-07-04:
Please note that the devel project now contains the "performance" module written by the author of this article. It provides a much more accurate measurement than the above patches, and require no patching of core. It also allows measurement of different URLs on the site.
Update 2012-09-19:
The patch has been ported to Drupal 7.x by Anton, and is attached to this article.
Attachment | Size |
---|---|
![]() | 1.62 KB |
![]() | 1.68 KB |
![]() | 21.54 KB |
![]() | 116 KB |
![]() | 7.82 KB |
![]() | 7.26 KB |
![]() | 5.38 KB |
Comments
Nice report!
Hi, nice reports! In my browser page aren't displayed correctly.. can you provide a PDF version or something that don't breaks line-returns?
I'd like to test your patches on my next-on-production website.. I'll let you know the results!
Bye!
Attached
Hi,
Thanks for planning to test this.
I attached the report in Open Office and Excel format. There are two sheets in that file, one for Drupal 5.1 (open buffet) vs. Drupal 6, and the other for a Drupal 5.2 minimal install vs. Drupal 6.
Let us know the results.
--
2bits -- Drupal consulting
Thanks for the drupal-5
Thanks for the drupal-5 patch, it made clear to me that some modules really aren't needed on a live site.
Like update_status and image_import.
re:
I'm not an expert in Drupal.But I think it do not take 3500mb.I also have around 35mb per page also, and thats with an opcode cache and memcache.I'd really like this discussion to go further.
Patch file permission issues
I just ran your 5.12 patch and notice the permissions were not set properly? It placed, on a windows system, everyone and none on the patched files when they should be set to System and admin under security. Is this by design or am I to set the permissions as I patch?
Drupal Module Memory and Http 500 Errors ...
http://www.nisekonews.com is a small Drupal 5.2 / G2 site with a few modules on shared hosting with 32M of memory. Every few weeks or so the site will stop working and only serve 500 errors. The host http://www.ezoshosting.com has found that they must increase the memory to 512M and then the error will go away. Once the error goes away they can decrease the memory back to 32M.
The sites current Drupal module memory footprint can be found at the footer of the site at
http://www.niskeonews.com (unless I need to remove it by the time you read this). The memory use today is posted below as well as some server info.
The server is RedHat, Apache 1.3.37, PHP 4.4.7, and MySQL 5.0.27-standard.
Drupal is drupal-5.2.tar.gz, brushed_steel-5.x-1.2.tar.gz, g2image-v3.0-RC1.zip, gallery-2.2.2-full.tar.gz, gallery-5.x-2.x-dev.tar.gz,
article-5.x-2.2.tar.gz, directory-5.x-2.0.tar.gz, google_analytics-5.x-1.x-dev.tar.gz, lightbox2-5.x-1.x-dev.tar.gz, lightbox2.03.3.zip, panels-5.x-1.2.tar.gz, scriptaculous-drupal.zip,
tinymce-5.x.tar.gz, tinymce_2_1_1_1.tgz, tinymce_compressor_php_1_1_0.tgz, views-5.x-1.6.tar.gz,
weather-3.x-3.1.tar.gz
The memory for today is below. I will be checking over time to see if there is a memory leak and where ...
Today,September 27, 2007, 8:30 am
Start,62784
End,29952112
Total,29889328
user,1195944
node,1143672
system,882304
views,814504
views_ui,808736
comment,702808
taxonomy,599240
filter,579432
weather,511392
tinymce,482080
forum,442168
directory,360624
upload,354640
panels,327368
gallery,311552
block,297248
menu,279088
poll,238240
statistics,226728
contact,215376
article,149968
blog,113848
lightbox2,97752
googleanalytics,94072
watchdog,92384
gallery_menu,78536
Modules,11399704
Bootstrap,18489624
Follow Up ...
I swapped the statistics module for the google analytic module and disabled the watchdog query and removed the watch dog module as well as employed the low memory patch from
http://wtanaka.com/drupal/system
My memory stats are now lower
Today,October 11, 2007, 12:20 pm
Start,63280
End,28220072
Total,28156792
user,1195128
node,1143672
views,814504
views_ui,808736
comment,702808
taxonomy,599264
filter,579432
weather,509032
tinymce,482080
forum,442168
directory,360624
panels,327368
gallery,311632
block,297360
menu,279088
poll,238240
contact,215376
system,165752
article,149912
blog,113848
lightbox2,97752
googleanalytics,94368
gallery_menu,78536
Modules,10006680
Bootstrap,18150112
Thanks for your help. If the problem continues I will investigate further but am disabling the module memory reporting for now.
Its ba aaack ...
Well the Ad module was turned on the 500 errors started popping up intermittenly ... I grabbed this memory foot print and log before disabling the Ad module ...
Apache error log ...
Its only three three files, mostly the index file, sometime the ad file, and maybe once the cron file ... cron set to run once per day
Cannot allocate memory: couldn't spawn child process: /drupal/index.php
Cannot allocate memory: couldn't spawn child process: /drupal/modules/ad/adserve.php
Cannot allocate memory: couldn't spawn child process: /drupal/cron.php
The memory was ...
Total,29500064
ad,1075000
After the ad module was off ...
Total,28662592
Just what I needed!
This is just what I needed!
I am on hostmonster (awardspace before that), and they give an error page each time I go over 32M.
Turns out all of the modules consume quit a bit of memory! Simplenews goes over 2M! Just for a newsletter subscription!
Does anybody know of a good host? I mean really good for a Drupal site? I am thinking nearly dedicated, but still a bit affordable.
I am considering 2mhost, they offer 64M.
Another question. If I host multiple site on a domain. How will that affect the memory or won't it?
How to install patch
How do I install the patch?
I know this is ages since
I know this is ages since this person asked, but here for anyone else looking is a link to a video tutorial:
http://drupal.org/node/132745
Evaluating VPS options for a
Evaluating VPS options for a Drupal app. This is super useful!
Nice post! Any ideas on how
Nice post!
Any ideas on how to modify the patch for D5 to work in a windows server 2003 configuration? Getting memory_get_usage(); as undefined.
Module memory usage in 5.11
Hi,
Thanks for your informative article. I'm trying to track down why my Drupal install is using huge amounts of RAM, and would love to know what resources each module is using.
Is it possible to port your patch to work with Drupal 5.11?
Worked with 5.11
The patches work on 5.11, but I did a much improved patch for 5.12 and 6.6, and attached them above.
--
2bits -- Drupal consulting
Output to a file
Hi,
I love your patch!
It would be interesting to write the output to a file though, so it can be used on an active site without disturbing the visitors.
Or is there another way to show it only to the admin?
working on 6.8?
Just applied the 6.6 patch on 6.8. Installed ok but I don't get anything outputted to the /tmp directory. Has there been a chane on 6.8 or am I doing something wrong? Thanks
6.8
Same issue here - no output in a 6.8 install
edit settings.php
You need to add the following to settings.php
$conf['memory_profiling'] = 1;
Great patch!
Hi In Drupal 5.15 the
Hi
In Drupal 5.15 the following problem appeared warning: Cannot use a scalar value as an array in /home/xxxxx/public_html/xxxx/includes/memory.inc on line 28.
What about memory.inc
What about memory.inc file...? :-)
Bootstrap
Hello
I applied Your memory patch to an installation (5.8) which has now 191 modules.
The results where very similiar to your example before.
Then I installed an APC cache on the machine.
The amount of memory used by modules for a single page sunk to very low amout see my example below.
The most of the memory is now consumed by bootstrap.
It didn't change.
Why is it so ?
Can I do something about it ?
To compare:
Your results from your example before
Bytes MB
Start memory 111448 0,106285095
End memory 51984944 49,57670593
Total used by Drupal 51873496 49,47042084
Total used by modules 40898088 39,00345612
Total used by Drupal bootstrap 10975408 10,46696472
My results with APC Cache:
Bytes MB
Start 74472 0.071
End 19437656 18.537
Total 19363184 18.466
Modules 1945296 1.855
Bootstrap 17417888 16.611
Now I am wondering what for is bootstrap using that much memory and can I reduce it somehow ?
Regards
Stefan
Patch breaks update.php
Great patch, Khalid! I just wanted to mention that on Drupal 6.11 the 6.6.memory.patch breaks update.php.
Has any progress been made on: 1) Porting this patch to 7.x & 2) Adding its functionality to devel.module?
Out of memory - database.mysqli.inc on line 323
Hi!
I'm using drupal 6.12 after applying your patch file the bootstrap and memory errors were disappeared, but I got a new error at database.mysqli.inc on line 323.
Please advice me on this.
Regards,
Lord.C
How get to work with drupal 6.13?
Hi I like to really try this out. I am trying to understand the whole php memory limit and usage of modules and performance issue with Drupal 6. This is the only source I found that gives the memory usage by module with is what I need. I download an run the patch command looks like to installed, add the line to the settings.php,and went to the site I am wanting to test. I surf several pages. But I am not get any information, no file in temp. I have Drupal 6.13 installed. I am doing something wrong? Please help I really like to use this. I even post info I get back for other to see. I am doing all this on my local development environment. Please help.
Huge bootstrap?
I have a site with quite a collection of modules, including UberCart. I'm getting a HUGE bootstrap code block of memory:
Total code 10615300
Bootstrap code 87952672
Block data 57304
Total data 92317016
Drupal 102932316
PHP 103215572
Any idea what could be contributing to this?
Thank you very much
Drupal desperate webmaster
Hello,
My website is using too much memory on Dreamhost 4000 MB VPS and suddenly shows many errors.
Can I use you patch on Drupal 5.19 to find what is the cause of huge memory usage ?
Do you have any advice to find what is the problem ? (I only have the minimum useful modules enabled)
Sorry, i am completely lost and desperate and don't know how to find the cause of this high memory usage and sudden hundreds of errors (thousand) everyday.
Thank you so much.
Still need help :-)
Hello,
Sorry to write you here but i really have a big problem.
Can you tell me if I can use your patch to check what are the modules causing problems on my Drupal install 5.19 ?
I ask you first because i broke everything once by trying Devel and wouldn't like that this happen again.
Thank you so much.
no csv file
I am not seeing a CSV file generated. Is there a command that needs to be run or does it take a certain amount of time to show up?
Where can I find the
Where can I find the memory.inc file?
Have a look at the drupal 6.6
Have a look at the drupal 6.6 patch, the code is in there.
But it would definitely be nice to see it in the latest patch.
it rocks
Thank you very much for your patch
I was searching too for the memory.inc and found it in drupal-memory-6.6.patch file
I use drupal 6.14
Your patch gives me eyes to see that drupal was called for some images (png), i will change my htaccess right now
Thanks Again
Help installing on drupal 6.13
I'd really appreciate it if someone could help me install this patch. I've created includes/memory.inc from the ver 6.6 patch, installed 6.14 ver patch and added config line to settings.php, however the module refuses to give any output to my temp directory. Any clues?
Thanks for the great patch. I
Thanks for the great patch. I ran into a problem, however, where the 6.14 patch doesn't include the memory.inc part (i got it from the 6.6 version). Otherwise perfect :)
Update for Drupal 6.16
Could someone please update the patch to work with Drupal 6.16?
I tried it and theme.inc keeps getting rejected, they changed a lot in there. :(
Update for patched Drupal 6,16
maybe what we would really need is a patch for (D6.16 + patched includes/bootstrap.inc) as per http://drupal.org/node/732064#comment-2788744
performance.module in devel project
There is already a way: install the devel module, and under it there is a performance module that does several things, including measure memory consumption for each page.
Files changed to drupal version 6.17
Hi There
First of all great work, It helped me a lot.
I would like to contribute a bit by sending you to the following link . This should do the work for Drupal 6.17. I am sorry I dod not attached the patch file as I am a bit short in time and I have no familiarity with the patches stuff.
Pressflow 6.19
I've added a version of this patch for Pressflow 6.19. I don't know if it works with vanilla Drupal or not--it may need to be massaged a bit.
http://drupal.org/node/844176#comment-3802302
Drupal 7
Any idea on how to modify this patch for Drupal 7?
D7 patch for measuring memory consumption
I've had to track down a memory hog, so in the process re-built this (or something resembling it) for D7.
You can find a patch vs D7.15 at http://telamenta.com/techarticle/tracking-memory-usage-drupal-7
I'm trying to fork drupal on github so I can put it there and simplify future patches. If that works you'll be able to find it here: https://github.com/asciikewl/drupal For the moment it is giving "Repository temporarily unavailable."
Thanks
Thanks for sharing this.
I added a link to your site at the bottom of article.
It seems to me that this is
It seems to me that this is still quite relevant for those looking to gain insight on what modules are using large amounts of memory. The peformance module is no longer part of Devel and the performance and logging module just gives summary data but no insight into potential problem modules. Is there an updated patch for the latest version of D6? Last update to the patch I saw was for Pressflow 6.19, anyone tried that on the latest vanilla Drupal. Or perhaps there's a better solution at this point?