Performance benchmarking of Drupal 5.12, Drupal 6.6, and Drupal 7.x: we are getting slower ...

Earlier this month, we published an article on benchmarking Drupal 5.x vs. 6.x: which one faster?

We wanted to take this analysis a step further and benchmark them both with Drupal 7.x as well.

Methodology

So, we got a checkout of Drupal 7.x as of October 24th, when update.php starting working for that version. We also used Drupal 5.12, and Drupal 6.6 which are the current and previous stable versions.

We started with the Drupal 5.12 site and using the devel module's generate feature, we created a site with the following data:

Users           4,950
Nodes           5,000
Comments    20,000
Vocabularies 10
Terms           1,000 

The front page had 30 nodes on it.

We selected the URLs of those same nodes and the home page itself for stress testing.

The following modules were enabled: system, block, color, comment, dblog, devel_generate, filter, help, menu, node, path, taxonomy, user, watchdog.

The exact Drupal 5 site was migrated to Drupal 6, and then that was migrated to Drupal 7. This ensured that the content is identical and is not a factor in any variations in results.

Benchmark Testing

We then used a test to stress the site for 2 minutes using 10 concurrent users. All the requests were completed successfully (HTTP code 200).

The tests were run with and without Drupal's page cache in normal mode, and with and without APC enabled.

They were run on the same machine in the same session. The server is a dual core AMD 64 X2 2.2GHz, 2GB RAM, 160GB SATA disk, with Ubuntu 8.04.1, Apache 2, MySQL 5.0, PHP 5.2.4.

Comparative data is what matters here, not absolute numbers.

Benchmark Results

      Description,   Trans,  Elap Time,  Resp Time, Trans Rate, Concurrent
Drupal 5; No APC; No cache,   1994,     120.25,       0.60,      16.58,       9.97
Drupal 6; No APC; No cache,   1787,     120.36,       0.67,      14.85,       9.96
Drupal 7; No APC; No cache,   1255,     120.47,       0.96,      10.42,       9.95
Drupal 5; No APC; w/ cache,  19481,     119.78,       0.06,     162.64,       9.98
Drupal 6; No APC; w/ cache,  15589,     120.41,       0.08,     129.47,       9.98
Drupal 7; No APC; w/ cache,  11418,     120.26,       0.11,      94.94,       9.98
Drupal 5; w/ APC; No cache,   6537,     120.42,       0.18,      54.29,       9.99
Drupal 6; w/ APC; No cache,   4706,     120.56,       0.26,      39.03,       9.97
Drupal 7; w/ APC; No cache,   2503,     120.58,       0.48,      20.76,       9.95
Drupal 5; w/ APC; w/ cache,  50698,     119.77,       0.02,     423.29,       9.96
Drupal 6; w/ APC; w/ cache,  46520,     120.49,       0.03,     386.09,       9.96
Drupal 7; w/ APC; w/ cache,  34359,     120.05,       0.03,     286.21,       9.98 

The columns are as follows:

  • Description: Description of the test.
  • Trans: Number of HTTP requests completed. 
  • Elap Time: Elapsed time, in seconds, taken to complete the tests. This should be close to 120 seconds, the duration set for the tests.
  • Resp Time: Response time per requests, in seconds across all requests.
  • Trans Rate: Number of transactions per second. The higher the better.
  • Concurrent: Number of concurrent users for the tests. This should be close to 10, as specified.

Analysis

Some quick calculations show that:

  • Without APC and no page cache, Drupal 6 is slower by 10.43% than Drupal 5, and Drupal 7 is slower by 29.83% than Drupal 6.
    • Drupal 7 is 37.15% slower than Drupal 5.
  • Without APC, but with page cache, Drupal 6 is slower by 20.39% than Drupal 5, and Drupal 7 is slower by 26.67% than Drupal 6.
    • Drupal 7 is 41.63% slower than Drupal 5.
  • With APC and no page, Drupal 6 is slower by 28.11% than Drupal 5, and Drupal 7 is slower by 46.81% than Drupal 6.
    • Drupal 7 is 61.76% slower than Drupal 5.
  • With APC, but with page cache, Drupal 6 is slower by 8.79% than Drupal 5, and Drupal 7 is slower by 25.87% than Drupal 6.
    • Drupal 7 is 32.38% slower than Drupal 5.

Again, there were no added views, and only the modules that are enabled by core, plus the path module.

Update 2008 October 29:

Here are some more data points to help us analyze the issue further.

This set of benchmarks is with Drupal page cache off, and MySQL query cache off as well, but with APC on.

    Resp Time   Trans Rate   Trans     OKAY    Failed   Elap Time   Concurrent
D5       0.26        38.94    4669     4669        0      119.89         9.98
D6       0.36        27.50    3314     3314        0      120.53         9.95
D7       0.92        10.82    1304     1304        0      120.49         9.97

The results themselves do not reveal anything new.

What is new is a set of spreadsheets that include the output of the performance logging module in summary and APC mode for each of the tests. 

This module is now part of the standard devel module, and available for Drupal 5.x, 6.x, and 7.x.

There is an OpenOffice version of the spreadsheet as well as an Excel version. Inside, you will find one sheet for each version.

Look at the columns for average query count and average milliseconds for query time.

The reason APC was left on is that this is the lowest overhead way of running the performance logging module without it impacting performance itself. The reason MySQL query cache is off is to not to mask any effect it has on the query time column.

Contents: 

Comments

Pardon if this is a dumb question...

What were the specific cache settings for drupal 6? Were the CSS/JS files compressed, and primed? I'd expect slower load times if they weren't and I'd also expect slower load times if they were being generated the first time when the test hit a random page.

No

No CSS nor JS compression was done in any of the tests.

Only the page cache was enabled or disable (the variable called 'cache' in Drupal).

The reason is that we are benchmarking just the page itself from the command line, not from a real browser that would request the JS, CSS or graphics components.

This was purely a comparative test for the backend, not any front end speedups.
--
2bits -- Drupal consulting

"Drupal 6 is slower by

"Drupal 6 is slower by 8.79%"

You give a result to three significant figures but not error margins? Sorry, but you failed the physics lab (first year, to behold). ;)

Lies, damn lies, and statistics

"Lies, damn lies, and statistics" is a famous saying which highlights the fallibility of using statistics. To that end...

It's reasonable to point out that a part of Drupal may have gotten slower in some situations, but the title of the article paints way to broad of a generalization, which unfortunately leaves too much room for grossly wrong assumptions and/or misinformation.

Lest there be any doubt - Drupal sites/versions continue to be more scalable, quicker, and able to serve more people than they ever have before. This is largely due to the growth of knowledge and/or contributed modules, but even core Drupal 6 tested 60+% faster than Drupal 5 in common situations which involve authenticated users - so next time, please not so fast with the broad statements on Drupal's speed that are based on such a specific set of circumstances!

Basic core is slowing down

Caleb.

You and me are talking about different things. I am focusing on core slowing down, while you are focusing on ways to scale Drupal despite that.

Of course I know that things can be done to speed up Drupal, I do this every week for clients. I also know we are getting better as APIs with every release. For example with Drupal 4.7, memcache was not possible or a hack. With Drupal 5.x, it needed core patches as well as patches to modules that have caches. With Drupal 6.x we don't have to do any of that, so installing memcache is far easier. The same goes for reverse proxy, which Drupal 6.x has support for.

Again, what I am talking about is basic core performance for common scenarios like viewing the front page (with only the login block appearing), and viewing individual nodes. This is a common-case as one could get.

What you are doing is essentially dismissing the issue as if it does not exist. I think a better approach is getting past the denial stage, to acceptance, and then moving to diagnosis and hopefully a set of remedies that would make core faster.

It is a common reaction to dismiss such figures when they come out. Just check the same reaction for Ubuntu getting slower, published later the same day I published the above.

See also what Andre Molnar said on his blog.
--
2bits -- Drupal consulting

I'm not an advanced drupal

I'm not an advanced drupal user, especially when it comes to system performance and fine tuning.

I'm not sure how drupal is "typically" used. I have dedicated linux server. I ran both 5.x and 6.x, with some minimum modules. My 5.x site typically gets a few authenticated visitors per day and a few more guests (I guess). I'm about to roll out a 6.x site, which I anticipate to have a few ten's of concurrent authenticated access at a burst.

Here are my non-quantitative observations: 6.x is way too slow compared to 5.x. I am too nervous to roll out my 6.x site to users. Aside from the excessive slow access of admin/build/modules (20+ seconds to over minutes), importing some cck types kills the session, either because too slow, or ran out of memory. I ended up setting execution time limit to 120 sec, and memory to 128M (it was 20M before). I need to demo to users tomorrow, and I found the memory problem today. I’m desperate and that’s the reason I’m googling on “drupal performance” at the moment.

I also have shared windows server. When I tried to install there a few days before, it simply times out left right, since I don't have any control, I just gave up.

I'm not bashing drupal. I vested too much in drupal to wish any evil. I'm just a nervous user.

Drupal obviously is capable of scaling up to huge site. drupal.org is an example. I wonder what kind of server(s) is (are) used to support drupal.org?

I assume my Linux server is tiny. My ISP says it’s Intel® Celeron® 1.8 GHz, 512 RAM. Is this too small to support drupal? It shouldn't be too bad compared with "typical" shared hosting, right? Where is the catch?

Not too much specifics

Well, your post does not have a lot of specifics in it to go by, apart from the site being Drupal 6.x.

The issue is that there is rarely any such thing as "Drupal". This is not Wordpress which is configured for one main thing (blogging) and not much else. Drupal sites are very customizable and there is a large selection of modules available to make the site into anything you want. So there is no standard way to setup a site, and hence troubleshooting has to tailor for each site's case.

Another issue is that with how sites are built nowadays, there are views all over the place with CCK up the wazoo. This causes lots of queries to be run for every page load for logged in users.

Add to that there is no code cache/accelerator, and you have issues in performance.

Your server's memory should be OK for such a site, of course the more the better. Celeron? Maybe yes, maybe no.

Before you try to solve the problem,  you have to diagnose it first.

Install the devel 6.x-1.x-dev module, and enable the performance module, and get a few statistics on page generation times, number of queries and time taken for queries. That should be a starting point on where the issue is, database, code or otherwise.

Thanks for the

Thanks for the response.

Right now, I'm concentrating on rolling out the site. I've found a link on tuning http://drupal.org/node/2601 I'll chew on it later. I really need to figure out what's going on.

I did install devel module and the query numbers look bad. I'll generate some reports and post it here later. But for my local server (2G RAM, Pentium 2GHz), I posted some data here http://drupal.org/node/311626

You know what, it dons on me that your current site runs on drupal. The performance looks excellent to me. What kind of server spec do you have? Is this site on 6.x?

I definitely will come back for more discussion after I roll out my stuff.

Don't post the full reports

Don't post the reports, since they will be just too big and ugly here. You can post the summary or conclusions of course.

This site is on 6.x, yes. It is on a 512MB VPS.

Please note that this discussion is about benchmarking Drupal 5.x vs. 6.x, vs. 7.x, and how to make 7.x better than what it is in a generic way.

You are delving into issues specific to your site's particular setup, and that is wandering offtopic.

Block Cache?

This was without the block cache enabled, correct?

A nice analysis (hint, hint) would be to identify where any major speed changes have occurred between different versions of core.

No.

There is only one block enabled, which is the login block.

So, the effect of that on these results is negligible.

Yes, if the site had 12 blocks that render from views, that would be heavy. But in this case, block caching or lack thereof plays no role.

And all the Drupal versions were benchmarked the same way to be apples to apples.
--
2bits -- Drupal consulting

That's pretty shocking

Any hints to where these slow downs come from?

Not yet.

No. All I can say is that is it in core somewhere, not due to contrib modules.

It takes a lot of time and effort to find where the slow downs are.

This would be with a code profiler.
--
2bits -- Drupal consulting

inaccessible

Your text is too small, and when I browser resize it it doesn't wrap but pushes over the faux column which is black background.

Yes, the same here (monitor

Yes, the same here (monitor 1024px, FF3).Tried to get letters biger but got part of the text on black background. It's hard to read your blog a little...

Jumping the shark

Drupal is getting slower and more complex. The 6th version is much slower due to bootstrap changes which are geared towards shared hosting users, while majority of Drupal installs are on VPS and dedicated servers (as shared hosting has 32M limit). With Drupal 7 including the database intensive registry patch and PDO I'll bet it'll be 2 times slower. Drupal is obviosly jumping the shark here with one year releases and slowing updates.

The one year timeframe between releases needs to be changed to two years, and more attention needs to be given to performance. Also, Drupal core developers should be more upfront about performance issues with new releases. For example, the 6th version advertised performance improvements, which seems dishonest in light of the benchmarks above.

No, no and no ...

Yes, core is getting slower under certain circumstances. We need to identify those and attempt to remedy them.

We cannot conclude that Drupal is "jumping the shark" and that the core developers are not upfront.

Drupal 6 is faster for sites that have many blocks (a very common use case), and supports caching (like memcache) out of the box without patching like we have to do with Drupal 5.x.

See my reply to Caleb below, and Andre Molnar's blog post linked in it.

Instead of just criticizing like that, please grab the new spreadsheets I posted today, and help analyze them in this discussion.
--
2bits -- Drupal consulting

Drupal 7 is a bit faster now

thanks for these benchmarks.

since this landed:

http://drupal.org/node/325665

Drupal 7 is a bit faster. would you mind rerunning the benchmarks to see if the gap closed by much?