Measuring memory consumption by Drupal bootstrap and modules

In a previous article, I described the "open buffet binge syndrome" where sites overconsume contrib modules, and gave examples where Apache memory was more than 100MB per process.

Today, I tried to take that further, and trace where this memory is consumed and by which modules. This lead to several interesting tangents.

I created a patch for Drupal 6 that can trace this information, and ran it on a replica of a live site with too many modules running on Drupal 5.1, to a default install of Drupal 6 HEAD (checked out Aug 27).

I then compared a plain default install of Drupal 6.x site to a plain Drupal 5.2 site, and then separated the amount of memory used by modules from the memory used by Drupal's bootstrap (inferred).

Note that this measurement is not the entire story, because depending on the underlying data and the exact page visited, memory consumption would be more due to loading/populating data structures.

Here is the table of the data, comparing a bloated Drupal 5.1 site to Drupal 6.x (Aug 27, 2007).

Update: I attached the results in Open Office and Excel format. There are two sheets in there. 

Drupal 5.1 bloated site vs. Drupal 6

As you can see, the number of modules can make the difference between using 11MB and 50MB!

Comparison of basic Drupal 5.2 and Drupal 6.x

I have not seen any significant difference between having APC on and having it disabled.

The patch for Drupal 6 and Drupal 5.2 are attached for those who want to use them.

Where does this go from here? Several things I like to see happen:

• The results of these tests need to be replicated by a few other people.
• The differences between Drupal 5.2 and Drupal 6 need to analyzed and see whether they warrant more changes. The split mode did improve the modules part, but the Drupal bootstrap part has worstened a bit.
• The includes/module.inc part to go into core, so measurement can be done without core modifications.
• The parts in index.php should move to the devel module, so measurement can be done by that module.

Thoughts? Comments?

Update 2008 October 22: I attached two new completely rearchitected patches, one for Drupal 5.12 and one for Drupal 6.6, both came out today.

The patch will not do anything until you add this configuration variable to your settings.php file:

$conf['memory_profiling'] = 1;

The patch now appends the results to a file called memory.csv in Drupal's temp directory, e.g. /tmp, or what you have set it to if different from the default.

The data in the file is not sorted, but can easily be loaded in OpenOffice or Excel and reformatted in any way you like.

Update 2009-09-30:

The re-architected patch has been rerolled for Drupal 6.14, the latest version for now.

Any help in porting this to Drupal 7.x, with the new registery is appreciated.

Update 2010-07-04:

Please note that the devel project now contains the "performance" module written by the author of this article. It provides a much more accurate measurement than the above patches, and require no patching of core. It also allows measurement of different URLs on the site.

Update 2012-09-19:

The patch has been ported to Drupal 7.x by Anton, and is attached to this article.