CLI Job Runner for XF 2.1 1.5.0

[FloV]

@Sim, are you planning to implement the proposed changes by @Xon?

 

[maxicep]

@Sim, are you planning to implement the proposed changes by @Xon?

Yes, it would be perfect if you made changes on your add-on @Sim .

 

[Sim]

I will look at it this week.

 

[Sim]

Sim updated CLI Job Runner with a new update entry:

v1.2.0

Thanks to @Xon for identifying the bug fix where AJAX JSON responses would still trigger job.php; and for making suggestions on avoiding unnecessary database queries and prevent running cron jobs unless triggered via our CLI job runner

• changes: disable calculations for job auto run time to avoid unnecessary database queries
• bug fix: disable AJAX auto job runner from triggering from AJAX JSON responses
• changes: don't allow cron tasks to execute unless...

Read the rest of this update entry...

 

[maxicep]

Thank you for update, sounds perfect. I will update Asap

 

[Sim]

@Sim there are a couple places you need to override to stop job.php from being polled.

Spoiler: Extend \XF\Job\Manager

class Manager extends XFCP_Manager
{
  public $allowCron = false;

  public function setAllowCron($value)
  {
      $this->allowCron = $value;
  }

  public function scheduleRunTimeUpdate()
  {
      if (empty(\XF::options()->svRunCronFromCron))
      {
          parent::scheduleRunTimeUpdate();
      }
  }

  public function getRunnable($manual)
  {
      if ($manual || $this->allowCron)
      {
          return parent::getRunnable($manual);
      }

      if (empty(\XF::options()->svRunCronFromCron))
      {
          return parent::getRunnable($manual);
      }

      return $this->db->fetchAll("
   SELECT *
   FROM xf_job
   WHERE trigger_date <= ?
      AND manual_execute = ?
       AND unique_key != 'cron'
   ORDER BY trigger_date
   LIMIT 1000
", [\XF::$time, $manual ? 1 : 0]);
  }
}

You want to extend \XF\Job\Manager::scheduleRunTimeUpdate to catch autoJobRun being updated to reduce unneeded DB writes.

Extending \XF\Job\Manager::getRunnable is needed because if a job was queued in a request, then XF sets various attributes in addDefaultJsonParams based on internal state (and not the template autoRun variable!) which means the cron job could be evaluated when job.php executes.

Then update the CLI runner command to call setAllowCron(true).

As for why you would want to block the cron job from being executed via job.php;
View attachment 212841
Before 12:00, standard xf web-based cron with autoRun being removed like your add-on does.
After 12:00, the above code changes in a private add-on is deployed.

This is the classic "thundering herd" problem, where a large number of requests wake up at once, all go to get a lock and then go back to sleep after most fail to get anything done.

Thanks @Xon - I've just release v1.2.0 which addresses the issues you identified.

I had indeed missed the fact that AJAX JSON responses could cause job.php to be called. Pretty simple fix to disable that - although calling job.php is still required for AutoBlocking jobs - since they change the UI and actually do need to be executed immediately (currently only used by the Approval Queue).

Good call on preventing the registry updates when calculating next run times - they are indeed unnecessary when we are using our cron triggered CLI job runner.

Finally - although the previous changes should in theory prevent cron tasks from running when other jobs are triggered (except via the Approval Queue), it's probably a safer approach to be more explicit and only allow it when triggered via the CLI job runner in any case - thanks for the suggestion!

 

[Skyrider]

I'm curious. Is there anyway I can see the debug output/input of the cronjob? Sometimes a specific cron job takes a long while finish, so it would be nice to know to see some kind of debug log as to see where it is "hanging". Without the quiet command line, it'll merely show "No more runnable jobs pending", but I assume that's what the quiet line hides.

 

[Sim]

Sim updated CLI Job Runner with a new update entry:

v1.3.1

New features:

• implemented debug logger for Job execution time tracking
• extend the XF\Job\Cron class with new run function which logs execution time for cron tasks
• added new Cli command and test job for testing purposes

Read the rest of this update entry...

 

[Sim]

I'm curious. Is there anyway I can see the debug output/input of the cronjob? Sometimes a specific cron job takes a long while finish, so it would be nice to know to see some kind of debug log as to see where it is "hanging". Without the quiet command line, it'll merely show "No more runnable jobs pending", but I assume that's what the quiet line hides.

@Skyrider - take a look at the new release v1.3.1 - I've added the ability to show some basic debugging information on the console when xf:run-jobs is run in Verbose (-v), Very Verbose (-vv) or Debug (-vvv) modes.

You can also add logging code to your custom Jobs and Cron tasks to be displayed on the console for debugging purposes - even on a production machine. See instructions at the bottom of the addon overview page.

 

[Skyrider]

Awesome, thanks! Wasn't expecting an entire update for this . Appreciate it.

 

[robdog]

Thanks for the latest updates. Really helpful and I can remove my hacks.

 

[Xon]

For the record; using systemd timers instead of cron;

Timer; /etc/systemd/system/xf-cron.timer

[Unit]
Description=Timer trigger for xf-cron service
Wants=network-online.target

[Timer]
OnBootSec=1min
OnUnitActiveSec=20sec

[Install]
WantedBy=multi-user.target

Service (that the timer triggers); /etc/systemd/system/xf-cron.service

[Unit]
Description=xf-cron service

[Service]
Type=oneshot
User=www-data
Group=www-data
ExecStart=/usr/bin/php -q /path/to/your/forum/root/cmd.php --quiet xf:run-jobs

Change www-data to the user your site runs as, and /path/to/your/forum/root/ as required

Configure system;

systemd daemon-reload
systemd enable xf-cron.timer
systemd start xf-cron.timer

 

[rdn]

-q /path/to/your/forum/root/cmd.php --quiet

-q and --quiet seems duplicate?

 

[TickTackk]

-q and --quiet seems duplicate?

-q is for PHP and --quiet is the Symfony Cli used by XenForo.

 

[rdn]

Thanks, but the addons also support -q right?

 

[TickTackk]

It does.

 

[rdn]

Push Notifications will be process via this addon/cron?

 

[Sim]

Push Notifications will be process via this addon/cron?

This addon just processes any and all jobs queued by the system - so if push notifications are sent via a job, then yes it will.

It doesn't add or change any functionality beyond disabling the browser based job runner and replacing it with a CLI based job runner that does exactly the same thing that the browser one did - run all the queued jobs.

 

[Xon]

Push Notifications are done as part of post notifications, which run via jobs if they take too long.

 

[rdn]

I hope there's an option to force Push Notification be run always via this addon/cli.
Thanks Sim and Xon.