Performance benchmarking of Drupal 5.12, Drupal 6.6, and Drupal 7.x: we are getting slower ...

Earlier this month, we published an article on benchmarking Drupal 5.x vs. 6.x: which one faster?

We wanted to take this analysis a step further and benchmark them both with Drupal 7.x as well.

Methodology

So, we got a checkout of Drupal 7.x as of October 24th, when update.php starting working for that version. We also used Drupal 5.12, and Drupal 6.6 which are the current and previous stable versions.

We started with the Drupal 5.12 site and using the devel module's generate feature, we created a site with the following data:

Users           4,950
Nodes           5,000
Comments    20,000
Vocabularies 10
Terms           1,000 

The front page had 30 nodes on it.

We selected the URLs of those same nodes and the home page itself for stress testing.

The following modules were enabled: system, block, color, comment, dblog, devel_generate, filter, help, menu, node, path, taxonomy, user, watchdog.

The exact Drupal 5 site was migrated to Drupal 6, and then that was migrated to Drupal 7. This ensured that the content is identical and is not a factor in any variations in results.

Benchmark Testing

We then used a test to stress the site for 2 minutes using 10 concurrent users. All the requests were completed successfully (HTTP code 200).

The tests were run with and without Drupal's page cache in normal mode, and with and without APC enabled.

They were run on the same machine in the same session. The server is a dual core AMD 64 X2 2.2GHz, 2GB RAM, 160GB SATA disk, with Ubuntu 8.04.1, Apache 2, MySQL 5.0, PHP 5.2.4.

Comparative data is what matters here, not absolute numbers.

Benchmark Results

      Description,   Trans,  Elap Time,  Resp Time, Trans Rate, Concurrent
Drupal 5; No APC; No cache,   1994,     120.25,       0.60,      16.58,       9.97
Drupal 6; No APC; No cache,   1787,     120.36,       0.67,      14.85,       9.96
Drupal 7; No APC; No cache,   1255,     120.47,       0.96,      10.42,       9.95
Drupal 5; No APC; w/ cache,  19481,     119.78,       0.06,     162.64,       9.98
Drupal 6; No APC; w/ cache,  15589,     120.41,       0.08,     129.47,       9.98
Drupal 7; No APC; w/ cache,  11418,     120.26,       0.11,      94.94,       9.98
Drupal 5; w/ APC; No cache,   6537,     120.42,       0.18,      54.29,       9.99
Drupal 6; w/ APC; No cache,   4706,     120.56,       0.26,      39.03,       9.97
Drupal 7; w/ APC; No cache,   2503,     120.58,       0.48,      20.76,       9.95
Drupal 5; w/ APC; w/ cache,  50698,     119.77,       0.02,     423.29,       9.96
Drupal 6; w/ APC; w/ cache,  46520,     120.49,       0.03,     386.09,       9.96
Drupal 7; w/ APC; w/ cache,  34359,     120.05,       0.03,     286.21,       9.98 

The columns are as follows:

• Description: Description of the test.
• Trans: Number of HTTP requests completed. 
• Elap Time: Elapsed time, in seconds, taken to complete the tests. This should be close to 120 seconds, the duration set for the tests.
• Resp Time: Response time per requests, in seconds across all requests.
• Trans Rate: Number of transactions per second. The higher the better.
• Concurrent: Number of concurrent users for the tests. This should be close to 10, as specified.

Analysis

Some quick calculations show that:

• Without APC and no page cache, Drupal 6 is slower by 10.43% than Drupal 5, and Drupal 7 is slower by 29.83% than Drupal 6.
  • Drupal 7 is 37.15% slower than Drupal 5.
• Without APC, but with page cache, Drupal 6 is slower by 20.39% than Drupal 5, and Drupal 7 is slower by 26.67% than Drupal 6.
  • Drupal 7 is 41.63% slower than Drupal 5.
• With APC and no page, Drupal 6 is slower by 28.11% than Drupal 5, and Drupal 7 is slower by 46.81% than Drupal 6.
  • Drupal 7 is 61.76% slower than Drupal 5.
• With APC, but with page cache, Drupal 6 is slower by 8.79% than Drupal 5, and Drupal 7 is slower by 25.87% than Drupal 6.
  • Drupal 7 is 32.38% slower than Drupal 5.

Again, there were no added views, and only the modules that are enabled by core, plus the path module.

Update 2008 October 29:

Here are some more data points to help us analyze the issue further.

This set of benchmarks is with Drupal page cache off, and MySQL query cache off as well, but with APC on.

    Resp Time   Trans Rate   Trans     OKAY    Failed   Elap Time   Concurrent
D5       0.26        38.94    4669     4669        0      119.89         9.98
D6       0.36        27.50    3314     3314        0      120.53         9.95
D7       0.92        10.82    1304     1304        0      120.49         9.97

The results themselves do not reveal anything new.

What is new is a set of spreadsheets that include the output of the performance logging module in summary and APC mode for each of the tests. 

This module is now part of the standard devel module, and available for Drupal 5.x, 6.x, and 7.x.

There is an OpenOffice version of the spreadsheet as well as an Excel version. Inside, you will find one sheet for each version.

Look at the columns for average query count and average milliseconds for query time.

The reason APC was left on is that this is the lowest overhead way of running the performance logging module without it impacting performance itself. The reason MySQL query cache is off is to not to mask any effect it has on the query time column.