CLI Job Runner for XF 2.1

CLI Job Runner for XF 2.1 1.5.0

No permission to download
I mean how often the job should be executed and for the --time= function.

I don't really have enough data yet to make any recommendations - and it will also depend largely on the size of your forum.

I suggest starting with the defaults and then see how that goes.

If your cron task is set to run every minute, and you specify no option for --time and don't add a jobMaxRunTime setting to your config file, then every minute it will process jobs for a maximum of 30 seconds in total, with each job given a maximum of 8 seconds to run (although it is up to the job to stop running - so individual jobs may take a little longer to complete).

I should write a simple interface to show the current status of the job queue so you can see how busy it is.

PS: you forgot the quotation mark at the beginning of the variable jobMaxRunTime

Thanks - fixed.
 
I think that would help a lot.

I don't really have time to do an admin GUI at the moment, so for now I've implemented a simple CLI tool to show a list of pending jobs.

Upgrade to v1.1.0 and then you can run the xf:show-jobs command to show a table of outstanding jobs.

Bash:
$ php cmd.php xf:show-jobs                       

2 pending jobs found

+----------------+-----------------+----------------------+----------------------+
| Key            | Class           | Next Run             | Last Run             |
+----------------+-----------------+----------------------+----------------------+
| cron           | XF\Job\Cron     | 11-Apr-2019 10:52:01 | 11-Apr-2019 10:52:31 |
| xfUpgradeCheck | XF:UpgradeCheck | 12-Apr-2019 00:12:21 | 10-Apr-2019 21:24:03 |
+----------------+-----------------+----------------------+----------------------+

The current time is: 11-Apr-2019 10:52:31 (UTC+10:00)
 
@Sim, first I want to say great work. However, I still can't seem to shut off the AJAX running of jobs when hitting job.php in the root public folder. Or is the AJAX still supposed to hit that file on a regular basis but the config param will simply exit the execution of that file?

Thank again.
 
@Sim, first I want to say great work. However, I still can't seem to shut off the AJAX running of jobs when hitting job.php in the root public folder. Or is the AJAX still supposed to hit that file on a regular basis but the config param will simply exit the execution of that file?

Thank again.

No, it shouldn't be hitting it at all - I use a code event listener to hook into the template engine and tell it not to include the ajax trigger in the output.

Do you have code event listeners disabled?

Are there any other addons you have installed which hook into the templater_global_data code event?
 
I though, it should be run every minute on active forums because for example if cronjob sets for every 5 min (*/5 * * * *) users getting the alerts delay.

So, if cron runs in every minute, then why this add-on required i don't understand that point.

Nice add-on anyway, thanks @Sim
 
I though, it should be run every minute on active forums because for example if cronjob sets for every 5 min (*/5 * * * *) users getting the alerts delay.

So, if cron runs in every minute, then why this add-on required i don't understand that point.

Nice add-on anyway, thanks @Sim

Yes, I do recommend running it every minute.

For an active, busy forum, this addon isn't really required - the built in ajax triggered job runner is pretty robust and should work fine.

This addon is useful if you have a low volume forum - for example it's new without much traffic yet, or perhaps a private forum - where there aren't necessarily visitors around all the time to trigger jobs.

It can also be useful if you prefer to have jobs run outside of the HTTP process - this addon will trigger them using the CLI instead.
 
It can also be useful if you prefer to have jobs run outside of the HTTP process - this addon will trigger them using the CLI instead.

I understand, thanks.

What is the advantage of this exactly?
Page load times or what ?

Because, after i switch to CLI cronjob, my server loads increasing every minutes due run the job command
Abnormal CPU usage continuously.

Ekran Resmi 2019-09-11 14.43.28.webp
 
Last edited:
Because, after i switch to CLI cronjob, my server loads increasing every minutes due run the job command
Abnormal CPU usage continuously.

Look at your DB CPU usage ... if your MySQL server is at 92.7% CPU, then every other component is going to be struggling to complete its tasks since they all have to wait until the DB responds to their queries.

Am I reading that right that you've got 45GB of virtual memory used by mysql (with 12GB reserved) - have you really allocated that much memory to your MySQL server? How big is your forum? How much RAM do you have in your server?

Your database server should be largely idling unless you have thousands of users online.

Have you tuned your MySQL server? This is a good place to start: https://github.com/major/MySQLTuner-perl

I also note that each of your php-fpm threads is using 1.2G of RAM - what are you running on your server? My XenForo fpm processes typically run at around 500MB each.

Also - check how many outstanding Jobs you have using the xf:show-jobs CLI command - if you have hundreds of jobs in the queue then the server is going to work quite hard until they clear.

One of my sites regularly has 400-500 users online at a time and my MySQL server typically idles along at around 3 - 5% CPU usage.

If you really do have a huge forum - then there are going to be a LOT of jobs running. If you used the built in AJAX job runner - then your fpm processes will be taking the load of running those jobs. By using my CLI runner - you've got them running in a separate process which should hopefully use less RAM and decrease wait times on your fpm processes.

How much this really matters for site performance, I don't know - I've not tested it.

Do you have server diagnostics which show the length of your php-fpm queue or how many php-fpm slow requests you're getting. It would be interesting to see how they change with and without the CLI job runner. I use NGINX Amplify for tracking these things and for tuning my fpm processes (it's free for up to 5 servers).
 
I have about 7m post, 600k threads and 3m member on board.
And server is 32core 72GB of RAM server

mysql also nginx, php-fpm server optimized .

Problem is already mysql cpu usage increasing every minute while running that cli cronjob...

It is very cool normally, but if cli cronjob runs, mysql cpu usage increasing dramatically together with cli cronjob process. (as i mentioned above post)

For example; that is normal time
Ekran Resmi 2019-09-12 03.11.16.png
 
How many outstanding jobs do you have?

If the number of outstanding jobs is actually increasing rather than dropping, it could be that running CLI job runner once per minute is not enough.
 
How many outstanding jobs do you have?

cmd.php xf:show-jobs output

1 pending jobs found

+------+-------------+----------------------+----------------------+
| Key | Class | Next Run | Last Run |
+------+-------------+----------------------+----------------------+
| cron | XF\Job\Cron | 12-Sep-2019 03:40:01 | 12-Sep-2019 03:36:01 |
+------+-------------+----------------------+----------------------+
 
You could try tuning the jobMaxRunTime to higher than the default of 8 seconds, and also set the maximum execution time to higher.

$config['jobMaxRunTime'] = 30; and use something like --time=45 when trigger the runner to allow it to run for l longer than 30 seconds.
 
Are you saying you did truncate it, or want to truncate it?

Either way, it shouldn't affect it now - there's only one job outstanding. Is it still generating high CPU usage when it runs?

I'm wondering whether clearing the entity cache might be causing an issue - although I can't imagine what - it's only around for one execution cycle, so not as if it should be huge?
 
Are you saying you did truncate it, or want to truncate it?

Either way, it shouldn't affect it now - there's only one job outstanding. Is it still generating high CPU usage when it runs?

I'm wondering whether clearing the entity cache might be causing an issue - although I can't imagine what - it's only around for one execution cycle, so not as if it should be huge?
I though, xenforo 2.1.3 has only one cronjob and it calls the jobs from xf_cron_entry
So no problem about that.

Problem is, load increasing while running the CLI cronjob (even all crons are default by xenforo)
I'm going to disable this add-on next couple days and look what changes on loads, mysql, php-fpm, mysql etc.
 
I though, xenforo 2.1.3 has only one cronjob and it calls the jobs from xf_cron_entry

XF 2.x has a job which trigger the cron tasks. It could be that those regular cron tasks are taking a significant amount of resources to execute, which is why you're seeing the CPU spike from the CLI job runner.

There can be many other jobs created by the system though - for example, bulk emails are sent using jobs. I did a quick count and there are over 50 different jobs that could be triggered by the system at various times, in addition to the cron job.

I'm going to disable this add-on next couple days and look what changes on loads, mysql, php-fpm, mysql etc.

You should look out for one particular fpm thread generating significant CPU usage for a short period - that is likely going to be the one processing the cron.

You might find that the high CPU usage is actually completely normal for a forum of your size and you just don't normally see it because it's hidden amongst all of the other fpm processing. The CLI just splits it out into a separate process which makes it more obvious.

I'd be interested to know exactly how long the CLI CPU usage stays high for each time it runs.
 
@Sim there are a couple places you need to override to stop job.php from being polled.

PHP:
class Manager extends XFCP_Manager
{
  public $allowCron = false;

  public function setAllowCron($value)
  {
      $this->allowCron = $value;
  }

  public function scheduleRunTimeUpdate()
  {
      if (empty(\XF::options()->svRunCronFromCron))
      {
          parent::scheduleRunTimeUpdate();
      }
  }

  public function getRunnable($manual)
  {
      if ($manual || $this->allowCron)
      {
          return parent::getRunnable($manual);
      }

      if (empty(\XF::options()->svRunCronFromCron))
      {
          return parent::getRunnable($manual);
      }

      return $this->db->fetchAll("
   SELECT *
   FROM xf_job
   WHERE trigger_date <= ?
      AND manual_execute = ? 
       AND unique_key != 'cron'
   ORDER BY trigger_date
   LIMIT 1000
", [\XF::$time, $manual ? 1 : 0]);
  }
}
You want to extend \XF\Job\Manager::scheduleRunTimeUpdate to catch autoJobRun being updated to reduce unneeded DB writes.

Extending \XF\Job\Manager::getRunnable is needed because if a job was queued in a request, then XF sets various attributes in addDefaultJsonParams based on internal state (and not the template autoRun variable!) which means the cron job could be evaluated when job.php executes.

Then update the CLI runner command to call setAllowCron(true).

As for why you would want to block the cron job from being executed via job.php;
1572175776019.webp
Before 12:00, standard xf web-based cron with autoRun being removed like your add-on does.
After 12:00, the above code changes in a private add-on is deployed.

This is the classic "thundering herd" problem, where a large number of requests wake up at once, all go to get a lock and then go back to sleep after most fail to get anything done.
 
Top Bottom