CLI Job Runner for XF 2.1 1.5.0

[Sim]

Sim submitted a new resource:

CLI Job Runner - Disables the browser triggered job runner and implements a CLI triggered job runner for Unix cron

This XenForo 2.0 addon disables the browser triggered job runner and implements a CLI triggered job runner for use with Unix cron.

Requirements

This addon requires PHP 5.4 or higher and only works on XenForo 2.0.x

Installation

Install as per normal addon installation.

Note: once this addon is installed and activated, scheduled tasks will no longer run - so completing the remaining installation steps is critical to ensure your forum continues to function normally.

First...

Read more about this resource...

 

[AdamSLevy]

I used this to run a CRON job that needed to read and write files in the Xenforo filesystem. I am using a Systemd timer instead of CRON. I found that if I don't run the command from the XenForo root directory, all of the relative file paths in the plugin end up being wrong. I recommend updating the documentation to advise users to not run the command like this: php /path/to/xenforo/cmd.php xf:run-jobs but instead something like cd /path/to/xenforo/ && php cmd.php xf:run-jobs.

In the case of using a systemd service and timer you can use the following to run it every minute.
xenforo-run-jobs.service

[Unit]
Description=Run Xenforo CRON jobs

[Service]
Type=oneshot
User=http
SuccessExitStatus=1
WorkingDirectory=/path/to/xenforo
ExecStart=/usr/bin/php cmd.php --time=59 xf:run-jobs

[Install]
WantedBy=multi-user.target



xenforo-run-jobs.timer

[Unit]
Description=Run Xenforo CRON jobs every minute

[Timer]
OnCalendar=*-*-* *:*:00
RandomizedDelaySec=10s
Persistent=true

[Install]
WantedBy=timers.target

 

[Sim]

Sim updated CLI Job Runner with a new update entry:

Re-release

Same version, new addon_id.

If you already have v1.0.0 installed, there is no need to upgrade. There is no new functionality in v1.0.0a. New installations should use this v1.0.0a version to ensure compatibility with future releases.

Due to the change in addon_id, upgrading from v1.0.0 to v1.0.0a is not supported - but there is no data stored with this addon, so simply uninstalling v1.0.0 before installing v1.0.0a will be sufficient.

Read the rest of this update entry...

 

[Xon]

Oh, how did I miss this. Another add-on I don't need to port!

 

[sbj]

@Sim can you explain for casual people for what we can use this addon? I assume you have other scripts and plugins running on your XF server which need to be run frequently, right? Like even at times when no users are online (which would be a browser trigger)?

And you say:

Note: once this addon is installed and activated, scheduled tasks will no longer run - so completing the remaining installation steps is critical to ensure your forum continues to function normally.

Does this mean, even after configured correctly, the XF crons in ACP won't run at their scheduled tasks' times and they will run at times whatever you configured in the cron tab for CLI?

 

[Sim]

@Sim can you explain for casual people for what we can use this addon? I assume you have other scripts and plugins running on your XF server which need to be run frequently, right? Like even at times when no users are online (which would be a browser trigger)?

Exactly.

I run a small, private forum for an organisation who have about 80 members (it's a business association - business people who all work in the same industry and run a not-for-profit organisation to educate people about their industry and promote the services their members can offer).

The forums sometimes get no traffic for weeks at a time, but I still need the regular newsletters and other cron-based activity to happen, even if there is nobody there to trigger the jobs by visiting the site.

It also allows us to be more flexible in defining how long each job is allowed to execute for, and how long we can execute multiple jobs for, before we stop and wait for the next trigger. The browser based trigger is, by necessity, constrained in how long it is allowed to execute for - something cron tasks have much more flexibility with. This could potentially help on sites with high volumes of users and posts where there are potentially a lot of jobs to be executed.

Note that the XF cron system and the XF job system are basically one and the same thing. Cron tasks are just jobs that are triggered at a specific time. Jobs are used for all sorts of behind-the-scenes processing of XF data, and so timely processing of them is important.

Does this mean, even after configured correctly, the XF crons in ACP won't run at their scheduled tasks' times and they will run at times whatever you configured in the cron tab for CLI?

No - I just mean that if you install this addon, but fail to set up the unix cron task ... you don't get any XF jobs or XF cron tasks executing at all. It's just a reminder - don't forget to set up the unix cron.

This addon does exactly 2 things:

1. disable the browser based job trigger
2. provide a cli command for triggering job execution

Installing it without using unix cron to call the cli command will break your forum.

Once the unix cron is set up to trigger the provided cli command, XF jobs will be processed and XF cron tasks will execute as normal.

 

[sbj]

Thank you for your detailed explanation.

I see, then a use for "casual" owners would be if they run a backup script or for example this addon where you want that your backup runs at 2-5 AM in the morning. Usually at those times there are no users online to trigger the cron but this addon would trigger that. Thank you.

 

[Xon]

@Sim you should probably run the following after each runQueue call;

// keep the memory limit down on long running jobs
\XF::triggerRunOnce();
\XF::app()-em()->clearEntityCache();

A bunch of the longer running XF2 CLI jobs do this

 

[Sim]

@Sim you should probably run the following after each runQueue call;

// keep the memory limit down on long running jobs
\XF::triggerRunOnce();
\XF::app()-em()->clearEntityCache();

A bunch of the longer running XF2 CLI jobs do this

I had a good look through the code and \XF::triggerRunOnce(); is already called for every job executed - see \XF\Job\Manager::runJobInternal - so I don't think there's any need for me to call it again?

However, I think clearing the Entity Cache after each job to keep memory usage under control is probably a good idea, so I'll add that in to the loop for the next version.

 

[robdog]

@Sim, is this still working with 2.1? It seems like they are calling a root level job.php file.

 

[Sim]

Haven't had a chance to test it yet - will try and look today

 

[Sim]

Sim updated CLI Job Runner with a new update entry:

v1.0.1

Minor update for XF2.0 and now tested on XF2.1

• we still want to run jobs if the user has disabled version checking in config
• clear entity cache each time we run a job, to keep memory usage under control

Read the rest of this update entry...

 

[Sim]

@Sim, is this still working with 2.1? It seems like they are calling a root level job.php file.

No real changes between v2.0 and v2.1 here - I've released a minor update for v1.0.1, should work on both versions.

 

[Sam9]

I run a small, private forum for an organisation who have about 80 members (it's a business association - business people who all work in the same industry and run a not-for-profit organisation to educate people about their industry and promote the services their members can offer).

The forums sometimes get no traffic for weeks at a time, but I still need the regular newsletters and other cron-based activity to happen, even if there is nobody there to trigger the jobs by visiting the site.

From the other side of the spectrum running a forum with thousands of active users at a time, moving the the job runner to CLI makes a lot more sense. This cuts down on superfluous web requests, allows finer tuning of when jobs are run and with custom time limits, and can assign a higher memory limit to the CLI php process.

 

[rdn]

Thank you so much!
This is valid right?

* * * * * /usr/local/bin/php /home/nginx/public/cmd.php --quiet --time=50 xf:run-jobs >/dev/null 2>&1

 

[rdn]

In your example, this should be:

$config['jobMaxRunTime'] = 8;

 

[Sim]

Thank you so much!
This is valid right?

* * * * * /usr/local/bin/php /home/nginx/public/cmd.php --quiet --time=50 xf:run-jobs >/dev/null 2>&1

Yes that will work.

You don't really need to do the whole >/dev/null 2>&1 thing, since the --quiet switch suppresses pretty much everything anyway - and I kind of want to know if anything is being written to stderr by my cron task, because it means that something went wrong, but it's fine if you choose to do it that way.

 

[Tealk]

Hello,

what is the recommended setting?

 

[Sim]

Hello,

what is the recommended setting?

The recommended setting for what?

 

[Tealk]

Sorry @Sim I wasn't specific.
I mean how often the job should be executed and for the --time= function.

PS: you forgot the quotation mark at the beginning of the variable jobMaxRunTime