Improving the performance of Drupal's cron by using the Elysia cron module

One great feature that Drupal has is the ability to make modules run certain tasks, often heavy ones, in the background at preset intervals. This can be achieved by a module implementing hook_cron.

Core uses this feature to index new content for the search module, ping module to notify remote sites of new content, fetch new release information from drupal.org, poll other sites for RSS feeds, and more.

Various contributed modules use this for various purposes, such as mailing out newsletters, cleaning up logs, synchronizing content with other servers/sites, and much more ...

Core Cron: All or None

This powerful core feature has some limitations though, such as:

  • All hook cron implementations for all modules are run at the same time, in sequence alphabetically or according to module weight.
  • When cron for one module is stuck, all modules following it will not be executed, and cron will not run again until 1 hour has passed.
  • There is no way to know which module is the one that caused the entire cron to get stuck. Moreover, there is no instrumentation information to know which cron hook takes the most time.

2bits.com have proposed core patches to report overcome the lack of instrumentation, by logging the information to the watchdog. The patches are useful only to those who apply them. It is unlikely that they will get in core any time soon.

For a practical example, you can use Job Queue with our own queue mail module to improve end user response time, and avoid timeouts due to sending a lot of emails. This scheme defers sending to when cron is run, and not when a user submits a node or a comment.

This works well, but for core, all cron hooks run at the same time. If you set cron to run every hour, then email sending could be delayed by an hour or even more if job queue cannot send them all in one run. If you make cron run more frequently, e.g. every 15 minutes, then all the heavy hooks such as search indexing and log cleanup will also run every 15 minutes consuming lots of resources.

Enter Elysia Cron ...

With Elysia cron, you can now have the best of both worlds: you can set cron for job_queue to run every minute, and defer other heavy stuff to once a day during off hours, or once an hour. The email is delivered quickly, within minutes, and we don't incur the penalty of long running cron hooks.

Features and Benefits of Elysia Cron

The features that Elysia cron offers are many, the important ones, with a focus on performance, are:

  • You can run different hook_cron implementations for different modules at a different frequency.
  • You are aware what the resource and performance impact of each hook_cron implementation is. This includes the time it took to run it last, the average, and maximum time. This information is very valuable in distributing different hooks across the day, and their frequencies.
  • Set a configurable maximum for cron invocations. Drupal core has a hard coded value of 240 seconds. You can adjust this up or down as per your needs.
  • Handles "stuck" crons better than core. In core, if cron is stuck, it takes one hour for it to automatically recover. In Elysia cron, the other hook invocations continue to run normally.
  • You can set the weight for each module, independent from the weight for the module in the system table. Using this, you can have a different order of execution for modules.
  • You can group modules in "contexts", assigning different run schedules for different contexts, or disable contexts globally.
  • The ability to assign a cron key, or a white list of allowed hosts that can execute cron.
  • Selectively disable cron for one or more modules, but not others, or all cron.
  • Selectively run cron for only one module.
  • Defining a cronapi that developers can use.
  • It requires no patching of core or contributed modules.

Examples of Elysia Cron in action

Here is captcha's cron, which has been configured to run only once a day in the early hours of the morning:

As well, dblog's cron runs once a day too. No need to trigger this every hours or twice an hour.

Search here is shown to be the most heavy of all cron hooks. But still, we run it twice every hour, so that the search is always fresh.

Statistics cleanup is kind of heavy too, so we run it only once a day.

Finally, xmlsitemap is a useful module, yet it is also heavy on a site with lots of nodes. Therefore we run it only once a day.

The above is not cast in stone for these module. They will vary from one site to the other depending on the server configuration, resources available and data set sizes. Moreover, even for the same site, it is recommended to monitor regularly and adjust these on an ongoing basis.

Alternatives to Elysia Cron

Elysia cron is no alone though. There are other modules that have overlapping functions, such as Super Cron, Cron Plus, and even a Cron API module. Super Cron seems promising, but Elysia does everything we need so far, so the motivation to evaluate it low on the list of priorities.

Here is an attempt to compare the various cron modules, but so far it is sparse on information.

A more powerful solution, but also more complex and heavy weight is the use of tools like Hudson Continuous Integration. Since it runs within Java it adds dependencies to the usual LAMP-only server as well as being more demanding on resources. You can read a full article on it here.

Contents: 

Tags: 

Comments

Saving my site from cron death!

When I was having some cron troubles, Elysia saved my bacon on more than one occasion. The module is quite elegant, and allows for fine-grained control over cron processes. For a smaller site, it really isn't necessary.

But for any site that either has maybe 40+ modules installed, gets over 1,000 visits/day, or has a lot of nodes, this module is essential.

Missing images?

Maybe I am a little dumb but it seems there is some content missing in the "Examples" section of this very useful article.

Images now showing

The images should be showing now.

Suggested settings for common tasks?

Great article! I came across Elysia a while back but wasn't quite sure how it compared with the other cron modules. I'll have to give it a closer look.

Are there any suggested settings for common cron tasks? Which to run hourly vs daily, etc.?

Also... the example images appear broken.

Each site is different

Well, each site is different. Depending on the modules you have, size of the site (number of nodes, comments, users), and traffic, things will be different.

Certain things should be very frequent, for example if you are using job_queue to defer mail to cron, then this should be every 1 or 5 minutes. That may need to change if you are using job_queue for other heavier stuff though.

Log cleaning tasks (statistics, dblog, ...etc.) should be once a day.

Content indexing is always a heavy tasks, but site owners always want fresh search indexes.

Hm, I get grumpy and don't

Hm, I get grumpy and don't want to touch any module that suggests I 'replace' core files with their own, even as just a suggestion.

You don't need to do that

Replacing cron.php is optional, not mandatory to use this module.

We use it with the standard core cron.php and it works well.

I think the module author is just documenting options, and perhaps exaggerating the impact of using the standard cron.php on performance. We have not seen any degradation so far.

One dumb question

I want to use Elysia Cron to run the search_cron once every hour and every 30 mts. session_expire (I want to use this module to clear the session clutter to improve the performance)

I've specified these intervals in Elysia Cron Settings but not able to figure out how Elysia runs itself? do I have to setup a Cron through the CPANEL for
http://www.example.com/sites/all/modules/elysia_cron/cron.php

and if yes, what is the frequency to call this?

Yes set a crontab... For

Yes set a crontab...

For example:
* * * * * wget -O - -q -t 1 http://www.example.com/sites/all/modules/elysia_cron/cron.php

Read the INSTALL.txt at module's folder for more info.