Earlier this month, we published an article on benchmarking Drupal 5.x vs. 6.x: which one faster?

We wanted to take this analysis a step further and benchmark them both with Drupal 7.x as well.

Methodology

So, we got a checkout of Drupal 7.x as of October 24th, when update.php starting working for that version. We also used Drupal 5.12, and Drupal 6.6 which are the current and previous stable versions.

We started with the Drupal 5.12 site and using the devel module's generate feature, we created a site with the following data:

Users           4,950
Nodes           5,000
Comments    20,000
Vocabularies 10
Terms           1,000 

The front page had 30 nodes on it.

We selected the URLs of those same nodes and the home page itself for stress testing.

The following modules were enabled: system, block, color, comment, dblog, devel_generate, filter, help, menu, node, path, taxonomy, user, watchdog.

The exact Drupal 5 site was migrated to Drupal 6, and then that was migrated to Drupal 7. This ensured that the content is identical and is not a factor in any variations in results.

Benchmark Testing

We then used a test to stress the site for 2 minutes using 10 concurrent users. All the requests were completed successfully (HTTP code 200).

The tests were run with and without Drupal's page cache in normal mode, and with and without APC enabled.

They were run on the same machine in the same session. The server is a dual core AMD 64 X2 2.2GHz, 2GB RAM, 160GB SATA disk, with Ubuntu 8.04.1, Apache 2, MySQL 5.0, PHP 5.2.4.

Comparative data is what matters here, not absolute numbers.

Benchmark Results

      Description,   Trans,  Elap Time,  Resp Time, Trans Rate, Concurrent
Drupal 5; No APC; No cache,   1994,     120.25,       0.60,      16.58,       9.97
Drupal 6; No APC; No cache,   1787,     120.36,       0.67,      14.85,       9.96
Drupal 7; No APC; No cache,   1255,     120.47,       0.96,      10.42,       9.95
Drupal 5; No APC; w/ cache,  19481,     119.78,       0.06,     162.64,       9.98
Drupal 6; No APC; w/ cache,  15589,     120.41,       0.08,     129.47,       9.98
Drupal 7; No APC; w/ cache,  11418,     120.26,       0.11,      94.94,       9.98
Drupal 5; w/ APC; No cache,   6537,     120.42,       0.18,      54.29,       9.99
Drupal 6; w/ APC; No cache,   4706,     120.56,       0.26,      39.03,       9.97
Drupal 7; w/ APC; No cache,   2503,     120.58,       0.48,      20.76,       9.95
Drupal 5; w/ APC; w/ cache,  50698,     119.77,       0.02,     423.29,       9.96
Drupal 6; w/ APC; w/ cache,  46520,     120.49,       0.03,     386.09,       9.96
Drupal 7; w/ APC; w/ cache,  34359,     120.05,       0.03,     286.21,       9.98 

The columns are as follows:

  • Description: Description of the test.
  • Trans: Number of HTTP requests completed. 
  • Elap Time: Elapsed time, in seconds, taken to complete the tests. This should be close to 120 seconds, the duration set for the tests.
  • Resp Time: Response time per requests, in seconds across all requests.
  • Trans Rate: Number of transactions per second. The higher the better.
  • Concurrent: Number of concurrent users for the tests. This should be close to 10, as specified.

Analysis

Some quick calculations show that:

  • Without APC and no page cache, Drupal 6 is slower by 10.43% than Drupal 5, and Drupal 7 is slower by 29.83% than Drupal 6.
    • Drupal 7 is 37.15% slower than Drupal 5.
  • Without APC, but with page cache, Drupal 6 is slower by 20.39% than Drupal 5, and Drupal 7 is slower by 26.67% than Drupal 6.
    • Drupal 7 is 41.63% slower than Drupal 5.
  • With APC and no page, Drupal 6 is slower by 28.11% than Drupal 5, and Drupal 7 is slower by 46.81% than Drupal 6.
    • Drupal 7 is 61.76% slower than Drupal 5.
  • With APC, but with page cache, Drupal 6 is slower by 8.79% than Drupal 5, and Drupal 7 is slower by 25.87% than Drupal 6.
    • Drupal 7 is 32.38% slower than Drupal 5.

Again, there were no added views, and only the modules that are enabled by core, plus the path module.

Update 2008 October 29:

Here are some more data points to help us analyze the issue further.

This set of benchmarks is with Drupal page cache off, and MySQL query cache off as well, but with APC on.

    Resp Time   Trans Rate   Trans     OKAY    Failed   Elap Time   Concurrent
D5       0.26        38.94    4669     4669        0      119.89         9.98
D6       0.36        27.50    3314     3314        0      120.53         9.95
D7       0.92        10.82    1304     1304        0      120.49         9.97

The results themselves do not reveal anything new.

What is new is a set of spreadsheets that include the output of the performance logging module in summary and APC mode for each of the tests. 

This module is now part of the standard devel module, and available for Drupal 5.x, 6.x, and 7.x.

There is an OpenOffice version of the spreadsheet as well as an Excel version. Inside, you will find one sheet for each version.

Look at the columns for average query count and average milliseconds for query time.

The reason APC was left on is that this is the lowest overhead way of running the performance logging module without it impacting performance itself. The reason MySQL query cache is off is to not to mask any effect it has on the query time column.

Comments

Mon, 2008/10/27 - 13:42

There is only one block enabled, which is the login block.

So, the effect of that on these results is negligible.

Yes, if the site had 12 blocks that render from views, that would be heavy. But in this case, block caching or lack thereof plays no role.

And all the Drupal versions were benchmarked the same way to be apples to apples.
--
2bits -- Drupal and Backdrop CMS consulting

Mon, 2008/10/27 - 11:09

Your text is too small, and when I browser resize it it doesn't wrap but pushes over the faux column which is black background.

Fri, 2009/03/20 - 05:12

Yes, the same here (monitor 1024px, FF3).Tried to get letters biger but got part of the text on black background. It's hard to read your blog a little...

Tue, 2008/10/28 - 21:02

Drupal is getting slower and more complex. The 6th version is much slower due to bootstrap changes which are geared towards shared hosting users, while majority of Drupal installs are on VPS and dedicated servers (as shared hosting has 32M limit). With Drupal 7 including the database intensive registry patch and PDO I'll bet it'll be 2 times slower. Drupal is obviosly jumping the shark here with one year releases and slowing updates.

The one year timeframe between releases needs to be changed to two years, and more attention needs to be given to performance. Also, Drupal core developers should be more upfront about performance issues with new releases. For example, the 6th version advertised performance improvements, which seems dishonest in light of the benchmarks above.

Wed, 2008/10/29 - 18:41

Yes, core is getting slower under certain circumstances. We need to identify those and attempt to remedy them.

We cannot conclude that Drupal is "jumping the shark" and that the core developers are not upfront.

Drupal 6 is faster for sites that have many blocks (a very common use case), and supports caching (like memcache) out of the box without patching like we have to do with Drupal 5.x.

See my reply to Caleb below, and Andre Molnar's blog post linked in it.

Instead of just criticizing like that, please grab the new spreadsheets I posted today, and help analyze them in this discussion.
--
2bits -- Drupal and Backdrop CMS consulting

Pages

Is your Drupal or Backdrop CMS site slow?
Is it suffering from server resources shortages?
Is it experiencing outages?
Contact us for Drupal or Backdrop CMS Performance Optimization and Tuning Consulting