Earlier this month, we published an article on benchmarking Drupal 5.x vs. 6.x: which one faster?
We wanted to take this analysis a step further and benchmark them both with Drupal 7.x as well.
Methodology
So, we got a checkout of Drupal 7.x as of October 24th, when update.php starting working for that version. We also used Drupal 5.12, and Drupal 6.6 which are the current and previous stable versions.
We started with the Drupal 5.12 site and using the devel module's generate feature, we created a site with the following data:
Users 4,950
Nodes 5,000
Comments 20,000
Vocabularies 10
Terms 1,000
The front page had 30 nodes on it.
We selected the URLs of those same nodes and the home page itself for stress testing.
The following modules were enabled: system, block, color, comment, dblog, devel_generate, filter, help, menu, node, path, taxonomy, user, watchdog.
The exact Drupal 5 site was migrated to Drupal 6, and then that was migrated to Drupal 7. This ensured that the content is identical and is not a factor in any variations in results.
Benchmark Testing
We then used a test to stress the site for 2 minutes using 10 concurrent users. All the requests were completed successfully (HTTP code 200).
The tests were run with and without Drupal's page cache in normal mode, and with and without APC enabled.
They were run on the same machine in the same session. The server is a dual core AMD 64 X2 2.2GHz, 2GB RAM, 160GB SATA disk, with Ubuntu 8.04.1, Apache 2, MySQL 5.0, PHP 5.2.4.
Comparative data is what matters here, not absolute numbers.
Benchmark Results
Description, Trans, Elap Time, Resp Time, Trans Rate, Concurrent
Drupal 5; No APC; No cache, 1994, 120.25, 0.60, 16.58, 9.97
Drupal 6; No APC; No cache, 1787, 120.36, 0.67, 14.85, 9.96
Drupal 7; No APC; No cache, 1255, 120.47, 0.96, 10.42, 9.95
Drupal 5; No APC; w/ cache, 19481, 119.78, 0.06, 162.64, 9.98
Drupal 6; No APC; w/ cache, 15589, 120.41, 0.08, 129.47, 9.98
Drupal 7; No APC; w/ cache, 11418, 120.26, 0.11, 94.94, 9.98
Drupal 5; w/ APC; No cache, 6537, 120.42, 0.18, 54.29, 9.99
Drupal 6; w/ APC; No cache, 4706, 120.56, 0.26, 39.03, 9.97
Drupal 7; w/ APC; No cache, 2503, 120.58, 0.48, 20.76, 9.95
Drupal 5; w/ APC; w/ cache, 50698, 119.77, 0.02, 423.29, 9.96
Drupal 6; w/ APC; w/ cache, 46520, 120.49, 0.03, 386.09, 9.96
Drupal 7; w/ APC; w/ cache, 34359, 120.05, 0.03, 286.21, 9.98
The columns are as follows:
- Description: Description of the test.
- Trans: Number of HTTP requests completed.
- Elap Time: Elapsed time, in seconds, taken to complete the tests. This should be close to 120 seconds, the duration set for the tests.
- Resp Time: Response time per requests, in seconds across all requests.
- Trans Rate: Number of transactions per second. The higher the better.
- Concurrent: Number of concurrent users for the tests. This should be close to 10, as specified.
Analysis
Some quick calculations show that:
- Without APC and no page cache, Drupal 6 is slower by 10.43% than Drupal 5, and Drupal 7 is slower by 29.83% than Drupal 6.
- Drupal 7 is 37.15% slower than Drupal 5.
- Without APC, but with page cache, Drupal 6 is slower by 20.39% than Drupal 5, and Drupal 7 is slower by 26.67% than Drupal 6.
- Drupal 7 is 41.63% slower than Drupal 5.
- With APC and no page, Drupal 6 is slower by 28.11% than Drupal 5, and Drupal 7 is slower by 46.81% than Drupal 6.
- Drupal 7 is 61.76% slower than Drupal 5.
- With APC, but with page cache, Drupal 6 is slower by 8.79% than Drupal 5, and Drupal 7 is slower by 25.87% than Drupal 6.
- Drupal 7 is 32.38% slower than Drupal 5.
Again, there were no added views, and only the modules that are enabled by core, plus the path module.
Update 2008 October 29:
Here are some more data points to help us analyze the issue further.
This set of benchmarks is with Drupal page cache off, and MySQL query cache off as well, but with APC on.
Resp Time Trans Rate Trans OKAY Failed Elap Time Concurrent
D5 0.26 38.94 4669 4669 0 119.89 9.98
D6 0.36 27.50 3314 3314 0 120.53 9.95
D7 0.92 10.82 1304 1304 0 120.49 9.97
The results themselves do not reveal anything new.
What is new is a set of spreadsheets that include the output of the performance logging module in summary and APC mode for each of the tests.
This module is now part of the standard devel module, and available for Drupal 5.x, 6.x, and 7.x.
There is an OpenOffice version of the spreadsheet as well as an Excel version. Inside, you will find one sheet for each version.
Look at the columns for average query count and average milliseconds for query time.
The reason APC was left on is that this is the lowest overhead way of running the performance logging module without it impacting performance itself. The reason MySQL query cache is off is to not to mask any effect it has on the query time column.
| Attachment | Size |
|---|---|
 benchmarking-drupal-5.12-vs-6.6-vs-7x.ods | 13.83 KB |
 benchmarking-drupal-5.12-vs-6.6-vs-7x.csv | 864 bytes |
| benchmarking-drupal-5.12-vs-6.6-vs-7x-per-page-details.ods | 20.78 KB |
| benchmarking-drupal-5.12-vs-6.6-vs-7x-per-page-details.xls | 122.5 KB |
Comments
Visitor (not verified)
Pardon if this is a dumb question...
Sun, 2008/10/26 - 23:44What were the specific cache settings for drupal 6? Were the CSS/JS files compressed, and primed? I'd expect slower load times if they weren't and I'd also expect slower load times if they were being generated the first time when the test hit a random page.
Khalid
No
Sun, 2008/10/26 - 23:54No CSS nor JS compression was done in any of the tests.
Only the page cache was enabled or disable (the variable called 'cache' in Drupal).
The reason is that we are benchmarking just the page itself from the command line, not from a real browser that would request the JS, CSS or graphics components.
This was purely a comparative test for the backend, not any front end speedups.
--
2bits -- Drupal and Backdrop CMS consulting
Visitor (not verified)
"Drupal 6 is slower by
Mon, 2008/10/27 - 01:35"Drupal 6 is slower by 8.79%"
You give a result to three significant figures but not error margins? Sorry, but you failed the physics lab (first year, to behold). ;)
Visitor (not verified)
Lies, damn lies, and statistics
Mon, 2008/10/27 - 02:13"Lies, damn lies, and statistics" is a famous saying which highlights the fallibility of using statistics. To that end...
It's reasonable to point out that a part of Drupal may have gotten slower in some situations, but the title of the article paints way to broad of a generalization, which unfortunately leaves too much room for grossly wrong assumptions and/or misinformation.
Lest there be any doubt - Drupal sites/versions continue to be more scalable, quicker, and able to serve more people than they ever have before. This is largely due to the growth of knowledge and/or contributed modules, but even core Drupal 6 tested 60+% faster than Drupal 5 in common situations which involve authenticated users - so next time, please not so fast with the broad statements on Drupal's speed that are based on such a specific set of circumstances!
Khalid
Basic core is slowing down
Wed, 2008/10/29 - 18:36Caleb.
You and me are talking about different things. I am focusing on core slowing down, while you are focusing on ways to scale Drupal despite that.
Of course I know that things can be done to speed up Drupal, I do this every week for clients. I also know we are getting better as APIs with every release. For example with Drupal 4.7, memcache was not possible or a hack. With Drupal 5.x, it needed core patches as well as patches to modules that have caches. With Drupal 6.x we don't have to do any of that, so installing memcache is far easier. The same goes for reverse proxy, which Drupal 6.x has support for.
Again, what I am talking about is basic core performance for common scenarios like viewing the front page (with only the login block appearing), and viewing individual nodes. This is a common-case as one could get.
What you are doing is essentially dismissing the issue as if it does not exist. I think a better approach is getting past the denial stage, to acceptance, and then moving to diagnosis and hopefully a set of remedies that would make core faster.
It is a common reaction to dismiss such figures when they come out. Just check the same reaction for Ubuntu getting slower, published later the same day I published the above.
See also what Andre Molnar said on his blog.
--
2bits -- Drupal and Backdrop CMS consulting
Visitor (not verified)
I'm not an advanced drupal
Sat, 2008/12/06 - 00:35I'm not an advanced drupal user, especially when it comes to system performance and fine tuning.
I'm not sure how drupal is "typically" used. I have dedicated linux server. I ran both 5.x and 6.x, with some minimum modules. My 5.x site typically gets a few authenticated visitors per day and a few more guests (I guess). I'm about to roll out a 6.x site, which I anticipate to have a few ten's of concurrent authenticated access at a burst.
Here are my non-quantitative observations: 6.x is way too slow compared to 5.x. I am too nervous to roll out my 6.x site to users. Aside from the excessive slow access of admin/build/modules (20+ seconds to over minutes), importing some cck types kills the session, either because too slow, or ran out of memory. I ended up setting execution time limit to 120 sec, and memory to 128M (it was 20M before). I need to demo to users tomorrow, and I found the memory problem today. I’m desperate and that’s the reason I’m googling on “drupal performance” at the moment.
I also have shared windows server. When I tried to install there a few days before, it simply times out left right, since I don't have any control, I just gave up.
I'm not bashing drupal. I vested too much in drupal to wish any evil. I'm just a nervous user.
Drupal obviously is capable of scaling up to huge site. drupal.org is an example. I wonder what kind of server(s) is (are) used to support drupal.org?
I assume my Linux server is tiny. My ISP says it’s Intel® Celeron® 1.8 GHz, 512 RAM. Is this too small to support drupal? It shouldn't be too bad compared with "typical" shared hosting, right? Where is the catch?
Khalid
Not too much specifics
Sat, 2008/12/06 - 00:46Well, your post does not have a lot of specifics in it to go by, apart from the site being Drupal 6.x.
The issue is that there is rarely any such thing as "Drupal". This is not Wordpress which is configured for one main thing (blogging) and not much else. Drupal sites are very customizable and there is a large selection of modules available to make the site into anything you want. So there is no standard way to setup a site, and hence troubleshooting has to tailor for each site's case.
Another issue is that with how sites are built nowadays, there are views all over the place with CCK up the wazoo. This causes lots of queries to be run for every page load for logged in users.
Add to that there is no code cache/accelerator, and you have issues in performance.
Your server's memory should be OK for such a site, of course the more the better. Celeron? Maybe yes, maybe no.
Before you try to solve the problem, you have to diagnose it first.
Install the devel 6.x-1.x-dev module, and enable the performance module, and get a few statistics on page generation times, number of queries and time taken for queries. That should be a starting point on where the issue is, database, code or otherwise.
Visitor (not verified)
Thanks for the
Sat, 2008/12/06 - 01:17Thanks for the response.
Right now, I'm concentrating on rolling out the site. I've found a link on tuning http://drupal.org/node/2601 I'll chew on it later. I really need to figure out what's going on.
I did install devel module and the query numbers look bad. I'll generate some reports and post it here later. But for my local server (2G RAM, Pentium 2GHz), I posted some data here http://drupal.org/node/311626
You know what, it dons on me that your current site runs on drupal. The performance looks excellent to me. What kind of server spec do you have? Is this site on 6.x?
I definitely will come back for more discussion after I roll out my stuff.
Khalid
Don't post the full reports
Sat, 2008/12/06 - 01:32Don't post the reports, since they will be just too big and ugly here. You can post the summary or conclusions of course.
This site is on 6.x, yes. It is on a 512MB VPS.
Please note that this discussion is about benchmarking Drupal 5.x vs. 6.x, vs. 7.x, and how to make 7.x better than what it is in a generic way.
You are delving into issues specific to your site's particular setup, and that is wandering offtopic.
Visitor (not verified)
Block Cache?
Mon, 2008/10/27 - 07:08This was without the block cache enabled, correct?
A nice analysis (hint, hint) would be to identify where any major speed changes have occurred between different versions of core.
Pages