XF 2.1 Cron entries

[squirrly]

About half my cron entries say the next run is sometime yesterday, which is obviously not correct/ideal. How do I resolve this?

 

[djbaxter]

As I understand it, cron jobs are triggered by page loads so on a fairly inactive forum it would not be unusual for a cron not to run on time. When this occurs, the cron job will run the next time someone visits the site to trigger it.

Xenforo code specialists please feel free to correct this if I'm wrong.

 

[squirrly]

But, that doesn't make sense - There are "daily" runs. How could the next Daily, or Hourly run be yesterday? They had accurate schedules in 1.5 yesterday; we upgraded to 2.1 last night and now the schedule looks ridiculous and I can't tell if they're not going to run or if they're just displaying inaccurately.

 

[squirrly]

Cron jobs that are scheduled should run at the scheduled time, not based on page loads.

 

[djbaxter]

How many daily visitors does your forum get?

 

[squirrly]

It's literally immaterial. Cron is a time-based scheduler. That's the point.

And, I'm not even complaining that jobs are or aren't running. I'm complaining that they're scheduled to run in the past, which doesn't work, and in my experience, if that's truly what is scheduled, it WILL break something.

I appreciate that you're responding, but I want this schedule to be accurate, and it isn't, and it needs to be accurate regardless of the traffic. Which - incidentally - hasn't changed in 24 hours.

 

[djbaxter]

XF 2.0 - What triggers cron jobs?

With vBulletin, there was a pixel in the footer that a visitor triggered which in turn triggered cron jobs to run. How does this work in Xenforo?

xenforo.com

deferred.php was the XF 1.5 mechanism.

In XF 2.x it uses job.php in the forum root directory - triggered by an Ajax call on certain page loads when the system detects jobs waiting to be run.

In XF 2.x, "jobs" are (typically) restartable processes which run in the background doing tasks which can sometimes take a long time to complete - it has built in mechanisms to process small chunks of work before stopping and then restarting again on the next trigger. Typically they are used for things like rebuilding data or caches, sending large batches of email, processing email bounces - all those background tasks which are not user-facing and aren't time-critical.

The cron system is actually one of these jobs!

The cron "job" looks through the list of cron entries and calculates which are due to run, then loops through and runs those cron tasks until the allowed execution time is reached - by default, that's 8 seconds and this is configurable via a config.php option. Note that it doesn't forcefully stop a task from running if it takes longer than 8 seconds, it just won't start any new tasks if it works out that it has already been running the various cron tasks for more than 8 seconds. It will stop and then wait to be triggered by the job manager again.

Normally a job will do it's work and then if it runs out of time, will update the system with counters for where it is up to and wait to be triggered again (waits for a "resume"). Alternatively, if it has finished its task (rebuilding data, or sending mail, or whatever), it will mark itself as "complete" and stop execution. Execution will start again (from the beginning) once explicitly triggered by some process or user/admin action.

The trick with the Cron job is that it never completes. The last step it takes, is to mark itself as unfinished and calculate the time the next cron task is due to run. It then sets that as the trigger time for job manager to run it again. The job entry for the cron system remains in the database - and will be run by the job system again once the trigger time passes.

If you look in the xf_job table on XF2.x, you should always see an entry for XF\Job\Cron - the trigger date will be the time it has calculated the next cron task is due to run. This is recalculated if you add or update a cron entry in the admin ui.

If you want a greater understanding of how it all hangs together - you can disable the browser based ajax trigger and trigger the job manager manually using my addon https://xenforo.com/community/resources/cli-job-runner.6478/ and watch what happens in the database as it does its thing.

That ajax call which triggers job.php will only appear on a page load if the system detects that there are jobs to be run.

If you look at the HTML tag at the top of the page - you'll see data-run-jobs="" appear whenever the system detects that there are jobs to be run:

<!DOCTYPE html>
<html id="XF" lang="en-US" dir="LTR"
    data-app="public"
    data-template="forum_list"
    data-container-key=""
    data-content-key=""
    data-logged-in="true"
    data-cookie-prefix="xf_"
    class="has-no-js template-forum_list"
     data-run-jobs=""> <!-- <== this is the magic bit! -->

This is picked up by the XF JavaScript which posts an Ajax call to job.php - search for XF.JobRunner in js/xf/core.js to see details.

 

[Chris D]

Cron jobs that are scheduled should run at the scheduled time, not based on page loads.

It has always been like this. A user visits the page and if there are outstanding jobs, an asynchronous request is made to job.php (previously deferred.php) which begins execution of the outstanding jobs. Crons are triggered like this a well. When it is triggered, it should execute any cron entries where the scheduled run time has passed.

On a site that has literally zero visitors, it is possible that the jobs (and cron entries) queue up until such a time there is a visitor that lands on the page and the jobs are triggered.

I assume you have more than zero visitors, though, so I'm going to assume that isn't the situation here.

The job running also relies on a database table to store the scheduled jobs.

1) Can you get the output of this query:

SELECT unique_key, trigger_date, last_run_date
FROM xf_job
ORDER BY trigger_date

2) Does an entry appear with a unique_key of cron?

 

[djbaxter]

@squirrly

I don't think you understand how PHP works. Any php script runs once only. To run it again, you need to reload the page.

These are not cron jobs running on your server. They are php scripts running from within the php-based Xenforo software.

 

[squirrly]

It has always been like this. A user visits the page and if there are outstanding jobs, an asynchronous request is made to job.php (previously deferred.php) which begins execution of the outstanding jobs. Crons are triggered like this a well. When it is triggered, it should execute any cron entries where the scheduled run time has passed.

On a site that has literally zero visitors, it is possible that the jobs (and cron entries) queue up until such a time there is a visitor that lands on the page and the jobs are triggered.

I assume you have more than zero visitors, though, so I'm going to assume that isn't the situation here.

The job running also relies on a database table to store the scheduled jobs.

1) Can you get the output of this query:

SELECT unique_key, trigger_date, last_run_date
FROM xf_job
ORDER BY trigger_date

2) Does an entry appear with a unique_key of cron?

Thanks. I can't run the query from my office, but I will when I get home. Should I input a ticket and link here, or continue with this thread for support?

We definitely have more than 0 traffic.

 

[squirrly]

@squirrly

I don't think you understand how PHP works. Any php script runs once only. To run it again, you need to reload the page.

These are not cron jobs running on your server. They are php scripts running from within the php-based Xenforo software.

Ok, cool. But since we have the same traffic as yesterday, telling me page views is the problem isn't an actual viable solution.

 

[Sim]

Ok, cool. But since we have the same traffic as yesterday, telling me page views is the problem isn't an actual viable solution.

No, but understanding how the cron system works is the first step to identifying the underlying issue! The XenForo cron system is not tied to the Unix cron system in any way.

I see a few potential causes:

1. your forum traffic is very low (ie a new site with few members) and thus the asynchronous ajax trigger is not being called enough
2. you have modified your themes and inadvertently removed the code which triggers the ajax calls
3. there is some kind of database or data corruption happening, causing the cron trigger to not find any data (eg the cron "job" is missing from the database as alluded to by @Chris D ) - I believe the cron job database entry (that triggers the cron tasks) can be recreated automatically by editing and saving any of the cron tasks in the admin UI.
4. there is a job which taking too long to complete (and then being re-queued for completion) and is thus effectively blocking other tasks from being completed

Understanding the contents of the xf_job table is important here - if that has a lot of data in it, then that indicates there is a backlog of tasks the system will need to get through before it can trigger the next cron job. Normally, there should only be a couple of jobs (including the "Cron" job).

 

[squirrly]

It has always been like this. A user visits the page and if there are outstanding jobs, an asynchronous request is made to job.php (previously deferred.php) which begins execution of the outstanding jobs. Crons are triggered like this a well. When it is triggered, it should execute any cron entries where the scheduled run time has passed.

On a site that has literally zero visitors, it is possible that the jobs (and cron entries) queue up until such a time there is a visitor that lands on the page and the jobs are triggered.

I assume you have more than zero visitors, though, so I'm going to assume that isn't the situation here.

The job running also relies on a database table to store the scheduled jobs.

1) Can you get the output of this query:

SELECT unique_key, trigger_date, last_run_date
FROM xf_job
ORDER BY trigger_date

2) Does an entry appear with a unique_key of cron?

There are 47,128 records, with the unique_key "null".

I think that's the same as the number of very, very old user accounts with 0 posts that we deleted via the batch tool. So, how do we clear this?

 

[squirrly]

1. your forum traffic is very low (ie a new site with few members) and thus the asynchronous ajax trigger is not being called enough
2. you have modified your themes and inadvertently removed the code which triggers the ajax calls
3. there is some kind of database or data corruption happening, causing the cron trigger to not find any data (eg the cron "job" is missing from the database as alluded to by @Chris D ) - I believe the cron job database entry (that triggers the cron tasks) can be recreated automatically by editing and saving any of the cron tasks in the admin UI.
4. there is a job which taking too long to complete (and then being re-queued for completion) and is thus effectively blocking other tasks from being completed

1. Forum is 15 years old. Has had 300+ views of the forum list today.
2. Themes are new but match what we had in test for several weeks, and no issues there.
3. I don't think that's it, because
4. There's 47k records in the table, either from a user cleanup OR the 1.5.7 --> 2.0.7 --> 2.1.3 upgrade which ran this time last night.

 

[Sim]

Is the number of records in the table decreasing?

 

[squirrly]

It's down about 60 records in the last almost 2 hours. Not down TO 60. . . It's maybe clearing 1 record every 90 seconds, average. At that rate, it'll take YEARS to clear them all.

 

[Sim]

There is a limit to how much processing a job can do with each request, so if you aren't getting a lot of requests to trigger jobs, it will take a while.

You could try installing my https://xenforo.com/community/resources/cli-job-runner.6478/ addon - this bypasses the browser-based job runner trigger and allows you to use real Unix crons to trigger jobs.

Having the jobs run every minute (regardless of visitor traffic) will hopefully get through the backlog much more quickly.

 

[squirrly]

It crept along and crept along until about 33000, and then zoomed through the rest and now it's down to 5 or less, and a ton of backed up emails are going out to folks.

 

[Groatz]

As @Sim says, CLI Job Runner FTW. Ideal for sites that have either very low visitor volume (and hence jobs not running), or very large sites (where it is detrimental to performance). Does require you to have a persistent server (ie not ephemeral) that you have ability to edit a crontab entry (doesn't need to be root, just as web server process owner). It's worked for our circumstances.

Regards, Groatz