Scheduling cron across many sites without stacking them

Every content management system suggests running its scheduled tasks at midnight. Follow that advice on thirty sites and you have built yourself a load spike at midnight. Scheduling across a network is a distribution problem, not a per-site setting.

What goes wrong when they stack

Cron jobs are not gentle. A backup reads the whole account, a feed importer makes external requests and writes rows, a sitemap rebuild walks every post. One of those at midnight is nothing. Twenty of them at midnight, inside accounts that share resource ceilings, is a queue.

The symptom is unmistakable once you know it: resource faults clustered at exactly the same minute every day, on a site with no traffic at that hour. That pattern is covered in Platform and Limits, and cron is its most common cause.

Give every site a different minute

The simplest workable scheme: assign each site a slot. Site one runs at seven past, site two at fourteen past, and so on. Nothing clever is required, and it removes the entire problem.

7 3 * * *   /usr/local/bin/php /home/acct/site1/cron.php
14 3 * * *  /usr/local/bin/php /home/acct/site2/cron.php
21 3 * * *  /usr/local/bin/php /home/acct/site3/cron.php

Avoid the top of the hour entirely. It is where every default lives, including things you did not schedule yourself.

Do not run everything every minute

A job scheduled every minute runs 1,440 times a day. If it takes two seconds and does nothing useful 1,430 of those times, you are paying for it constantly. Ask what the task actually requires:

• Publishing scheduled posts: every fifteen minutes is almost always fine.
• Feed imports: hourly, usually less.
• Sitemaps and caches: daily.
• Backups: daily, off peak, staggered.

Stop the job overlapping itself

If a job takes longer than its interval, the next run starts while the first is still going. Two copies compete, both slow down, and a third joins. On a shared ceiling this escalates quickly.

Use a lock file, or make the interval comfortably longer than the worst case runtime. If you do not know the worst case runtime, that is the thing to measure first.

Silence the successful ones

Cron mails output by default. A job that prints one line on success generates a mail every run, which fills a mailbox nobody reads and hides the failure messages you would actually want. Redirect normal output and keep the errors:

30 4 * * * /usr/local/bin/php /home/acct/site/task.php > /dev/null

Check what is already scheduled

On an inherited network, before adding anything, list existing jobs on every account. Duplicates from a previous setup, jobs pointing at deleted paths, and importers nobody remembers are common. Removing dead jobs is often the cheapest performance work available.

The cron interface is documented at docs.cpanel.net. For the wider routines of keeping many sites healthy, see Running Your Estate.