CLI Job / Cron Runner

CLI Job / Cron Runner 2.1.0

No permission to download

Sim

Well-known member
Sim submitted a new resource:

CLI Job Runner for XF 2.2 - An advanced CLI triggered job runner for Unix cron with extensive debugging support

This XenForo 2.2 addon implements an advanced CLI triggered job runner for use with Unix cron as a replacement for the built-in xf:run-jobs command introduced in XF 2.2

The command is called hg:run-jobs and does the same thing that xf:run-jobs does, but with more options and functionality.

Requirements

This addon requires PHP 7.0 or higher and has been tested on XenForo 2.2.0 beta 1

Installation...

Read more about this resource...
 

Sim

Well-known member
Sim updated CLI Job Runner for XF 2.2 with a new update entry:

v2.0.0b3 update - contains breaking changes!

v2.2.0b3 makes some back-end changes to the way we manage the lock files and fixes an issue where long running jobs could see multiple job runners executing at the same time.

This version has also been updated to more closely match some of the structure and options for the core job runner included in XF 2.2

Note breaking change: the --time option has been renamed --max-execution-time inline with the options provided by the XF2.2 core job runner.

Other...

Read the rest of this update entry...
 

Robert9

Well-known member
Thank you very much for that addon; the half day i i could not start my cron-jobs while the forum is on a small NUC here in my office and has just one user, so no triggering. FInally you have the solution (... i hope and try now ...)
 
  • Like
Reactions: Sim

Robert9

Well-known member
From today, you are my personal hero here. :)

But please tell me:

Your add-on installed.
Cron-Job Add-on installed
Cron-job with crontab -e added
Method for logging added to the job/file

Is this correct?

if ($this->doLogger == 1)
{

$this->log("message", ['title' => $title, 'message' => $message]);
}

($title and $message are two vars that should be send to the forum by API.)


...

Should this be in the admin log; there i see the cronjob and some html garbage.
Or in the server-log?
Or in your own log? I dont see any.
 
Last edited:

Robert9

Well-known member
Do you think it is possible that the server based crons can kill my xdebug?
With normal xf cron, i can "breakpoints" in cron/file, then it jumps to job/file;
with server based cron, it "breakpoints" in cron/file; then no more stops, while the job is done (sending a thread via API to the forum)
Strange ...
 

Sim

Well-known member
From today, you are my personal hero here. :)

But please tell me:

Your add-on installed.
Cron-Job Add-on installed
Cron-job with crontab -e added
Method for logging added to the job/file

Is this correct?

if ($this->doLogger == 1)
{

$this->log("message", ['title' => $title, 'message' => $message]);
}

($title and $message are two vars that should be send to the forum by API.)


...

Should this be in the admin log; there i see the cronjob and some html garbage.
Or in the server-log?
Or in your own log? I dont see any.

Okay, there's a couple of things happening here.

First up, the CLI job runner will output information to the console rather than to log files.

We normally run the Job Runner via Unix cron with the -q flag to suppress all unnecessary output - so you won't see any logging.

You would need to change the flags to increase the verbosity level:
  • Quiet: cmd.php -q hg:run-jobs
  • Normal: cmd.php hg:run-jobs
  • Verbose: cmd.php -v hg:run-jobs
  • Very Verbose: cmd.php -vv hg:run-jobs
  • Debug: cmd.php -vvv hg:run-jobs
However, even with all that turned on - when triggered via Unix cron, you won't see this output - unless your Unix cron system is configured to send you emails with the contents of any output generated (which can lead to you receiving a LOT of emails from your server!!).

If you are debugging a Job or XF Cron task on a dev server, I suggest you simply run the Job Runner manually from your console - don't trigger it via Unix cron. That way you'll see all output based on the verbosity level you specify on the command line.

If you want to run a specific Cron task manually, Beta 3 introduced two new commands (which I haven't really documented yet) - see below.

If you're trying to debug an issue with a Job running in production then you can force the Unix cron to output all information to a file:

Bash:
cmd.php -vvv hg:run-jobs --time=180 > /var/log/xenforo/`date +\%Y\%m\%d\%H\%M\%S`-cron.log 2>&1

... the above command will store all output in a file with the filename corresponding to the time the task was executed, in the /var/log/xenforo directory - you should make sure this path exists and that it is writable by the unix user that your cron task executes as.

The new commands for working with XF cron tasks are:

Show Crons

The hg:show-crons command will list all of your active XF cron tasks

Bash:
$ cmd.php hg:show-crons

19 active cron entries found

+---------------------------+----------------------+--------------------------+
| ID                        | Next Run (UTC+10:00) | Addon                    |
+---------------------------+----------------------+--------------------------+
| forumStatistics           | 29-Sep-2020 06:53    | XF                       |
| emailBounce               | 29-Sep-2020 06:53    | XF                       |
| warningExpiry             | 29-Sep-2020 06:55    | XF                       |
| rebuildSearchForumCache   | 29-Sep-2020 07:00    | XF                       |
| memberStatsCache          | 29-Sep-2020 07:00    | XF                       |
| feeder                    | 29-Sep-2020 07:02    | XF                       |
| cleanUpHourly             | 29-Sep-2020 07:10    | XF                       |
| emailUnsubscribe          | 29-Sep-2020 07:13    | XF                       |
| userGroupPromotions       | 29-Sep-2020 07:20    | XF                       |
| views                     | 29-Sep-2020 07:30    | XF                       |
| trophy                    | 29-Sep-2020 07:40    | XF                       |
| expireTempUserChanges     | 29-Sep-2020 07:42    | XF                       |
| deleteExpiredBans         | 29-Sep-2020 07:45    | XF                       |
| downgradeExpired          | 29-Sep-2020 07:50    | XF                       |
| fileCheck                 | 29-Sep-2020 10:10    | XF                       |
| dailyStats                | 29-Sep-2020 10:30    | XF                       |
| cleanUpDaily              | 29-Sep-2020 13:00    | XF                       |
| activitySummaryEmail      | 30-Sep-2020 00:20    | XF                       |
| sitemap                   | 30-Sep-2020 15:37    | XF                       |
+---------------------------+----------------------+--------------------------+

The current time is: 29-Sep-2020 06:52:05 (UTC+10:00)

There are three command line options you can specify:
  • -a or --all shows all cron tasks, including disabled tasks
  • -m or --method includes the class::method for each cron task
  • -s or --sort specifies the sort column for the list (date, id or addon)
You can use this tool to quickly identify the ID of the cron task you want to run - which you then use on the other new command ...

Run Cron

The hg:run-cron command will execute an XF cron task - simply specify a cron ID as the argument.

For example: cmd.php hg:run-cron activitySummaryEmail will run the "Send activity summary email" cron task.

By default, disabled cron tasks cannot be run - so you can override this by specifying the -f or --force option on the command line.

The same verbosity flags work here - so specify -v, -vv or -vvv to generate the desired level of output.
 

Sim

Well-known member
Do you think it is possible that the server based crons can kill my xdebug?
With normal xf cron, i can "breakpoints" in cron/file, then it jumps to job/file;
with server based cron, it "breakpoints" in cron/file; then no more stops, while the job is done (sending a thread via API to the forum)
Strange ...

Normally you would use a browser plugin to trigger the Xdebug session - but we're running in the CLI, so that's not going to work.

You should read up on how to trigger Xdebug sessions from the CLI.

If you've configured your php.ini to enable Xdebug - be aware that some unix distributions have a separate php.ini for the CLI, so you'll need to configure it there too.

If you are using PHPStorm - there are instructions for debugging CLI commands here: https://www.jetbrains.com/help/phpstorm/debugging-a-php-cli-script.html
 

Robert9

Well-known member
No. I have started the cron in the acp; and breakpoints in cron/file were reached, but then when calling job/file none of the breakpoints was called while the code itself was used to send a new thread to the forum.

I got the idea to start the cron by console. Then this code will not write anything to a logfile?

// My code
Code:
$this->log("Test", [
        'title' => $title,
        'message' => $message,
        'tags' => $tag,
        'prefix_id' => $prefixId,
        'statusCode' => $statusCode,
    ]
);

...

// your code
protected function log($message, array $context = [])
{
    // check to see if we actually have a logger available and abort if not
    if (!isset($this->app['cli.logger'])) return;

    /** @var Logger $logger */
    $logger = $this->app['cli.logger'];
    $logger->logJobProgress($message, $context, $this);
}
 

Sim

Well-known member
Exactly how did you start the cron? What command and options did you specify?

The Job Runner doesn't log output to a file - it outputs to the console, so you need to redirect output to a file using > on the command line if you want to save it.
 

Robert9

Well-known member
When i debug with storm/xdebug i start the cron with a click in the acp;
and i do it with the normal cron, not the server cron, because it else behaves strange as described above.

Normally i would await that the code for the logging would add the result to a log-file.
Later at the day i will try to use the admin-log for this; i hope this is only a question of some lines to add to job/file
 

Sim

Well-known member
Note that after installing the latest version, if you are seeing the message "There are scheduled jobs outstanding which have not run. Jobs may not be getting triggered when expected."

1601531153450.png

... check to see whether you are still using the --time command line option.

This option has been renamed to --max-execution-time to match the option names used in the core job runner.

Make sure you have updated all of your cron tasks to use --max-execution-time
 

VersoBit

Active member
When using this tool we are getting the following error:

[E_WARNING] fstat() expects parameter 1 to be resource, int given
  • 4 minutes ago
  • src/vendor/league/flysystem/src/Util.php:274

Code:
[2020-10-18 23:06:52] Hampel\JobRunner\Util\Lock: writing lock {"expiryTime":1603063042,"expiryTime_formatted":"2020-10-
18 23:17:22 UTC"} {}


In Util.php line 274:



[ErrorException]

[E_WARNING] fstat() expects parameter 1 to be resource, int given





Exception trace:

at /<path_to_forum>/src/vendor/league/flysystem/src/Util.php:274

XF::handlePhpError() at n/a:n/a

fstat() at /<path_to_forum>/src/vendor/league/flysystem/src/Util.php:274

League\Flysystem\Util::getStreamSize() at /<path_to_forum>/src/addons/XFAws/_vendor/league/flysystem-aws-s3-v3/src/AwsS3Adapter.php:588

League\Flysystem\AwsS3v3\AwsS3Adapter->upload() at /<path_to_forum>/src/addons/XFAws/_vendor/league/flysystem-aws-s3-v3/src/AwsS3Adapter.php:133

League\Flysystem\AwsS3v3\AwsS3Adapter->write() at /<path_to_forum>/src/vendor/league/flysystem/src/Filesystem.php:69

League\Flysystem\Filesystem->write() at n/a:n/a

call_user_func_array() at /<path_to_forum>/src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php:431

League\Flysystem\EventableFilesystem\EventableFilesystem->callFilesystemMethod() at /<path_to_forum>/src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php:395

League\Flysystem\EventableFilesystem\EventableFilesystem->delegateMethodCall() at /<path_to_forum>/src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php:247

League\Flysystem\EventableFilesystem\EventableFilesystem->write() at /<path_to_forum>/src/vendor/league/flysystem/src/MountManager.php:443

League\Flysystem\MountManager->write() at /<path_to_forum>/src/addons/Hampel/JobRunner/Util/Lock.php:51

Hampel\JobRunner\Util\Lock->write() at /<path_to_forum>/src/addons/Hampel/JobRunner/Util/Lock.php:30

Hampel\JobRunner\Util\Lock->get() at /<path_to_forum>/src/addons/Hampel/JobRunner/SubContainer/JobRunner.php:48

Hampel\JobRunner\SubContainer\JobRunner->getLock() at /<path_to_forum>/src/addons/Hampel/JobRunner/Cli/Command/RunJobs.php:58

Hampel\JobRunner\Cli\Command\RunJobs->execute() at /<path_to_forum>/src/vendor/symfony/console/Command/Command.php:255

Symfony\Component\Console\Command\Command->run() at /<path_to_forum>/src/vendor/symfony/console/Application.php:992

Symfony\Component\Console\Application->doRunCommand() at /<path_to_forum>/src/vendor/symfony/console/Application.php:255

Symfony\Component\Console\Application->doRun() at /<path_to_forum>/src/vendor/symfony/console/Application.php:148

Symfony\Component\Console\Application->run() at /<path_to_forum>/src/XF/Cli/Runner.php:109

XF\Cli\Runner->run() at /<path_to_forum>/cmd.php:15


hg:run-jobs [-t|--max-execution-time MAX-EXECUTION-TIME] [--manual-only]

Any ideas?
 

Sim

Well-known member
When using this tool we are getting the following error:

[E_WARNING] fstat() expects parameter 1 to be resource, int given
  • 4 minutes ago
  • src/vendor/league/flysystem/src/Util.php:274

Code:
[2020-10-18 23:06:52] Hampel\JobRunner\Util\Lock: writing lock {"expiryTime":1603063042,"expiryTime_formatted":"2020-10-
18 23:17:22 UTC"} {}


In Util.php line 274:



[ErrorException]

[E_WARNING] fstat() expects parameter 1 to be resource, int given





Exception trace:

at /<path_to_forum>/src/vendor/league/flysystem/src/Util.php:274

XF::handlePhpError() at n/a:n/a

fstat() at /<path_to_forum>/src/vendor/league/flysystem/src/Util.php:274

League\Flysystem\Util::getStreamSize() at /<path_to_forum>/src/addons/XFAws/_vendor/league/flysystem-aws-s3-v3/src/AwsS3Adapter.php:588

League\Flysystem\AwsS3v3\AwsS3Adapter->upload() at /<path_to_forum>/src/addons/XFAws/_vendor/league/flysystem-aws-s3-v3/src/AwsS3Adapter.php:133

League\Flysystem\AwsS3v3\AwsS3Adapter->write() at /<path_to_forum>/src/vendor/league/flysystem/src/Filesystem.php:69

League\Flysystem\Filesystem->write() at n/a:n/a

call_user_func_array() at /<path_to_forum>/src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php:431

League\Flysystem\EventableFilesystem\EventableFilesystem->callFilesystemMethod() at /<path_to_forum>/src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php:395

League\Flysystem\EventableFilesystem\EventableFilesystem->delegateMethodCall() at /<path_to_forum>/src/vendor/league/flysystem-eventable-filesystem/src/EventableFilesystem.php:247

League\Flysystem\EventableFilesystem\EventableFilesystem->write() at /<path_to_forum>/src/vendor/league/flysystem/src/MountManager.php:443

League\Flysystem\MountManager->write() at /<path_to_forum>/src/addons/Hampel/JobRunner/Util/Lock.php:51

Hampel\JobRunner\Util\Lock->write() at /<path_to_forum>/src/addons/Hampel/JobRunner/Util/Lock.php:30

Hampel\JobRunner\Util\Lock->get() at /<path_to_forum>/src/addons/Hampel/JobRunner/SubContainer/JobRunner.php:48

Hampel\JobRunner\SubContainer\JobRunner->getLock() at /<path_to_forum>/src/addons/Hampel/JobRunner/Cli/Command/RunJobs.php:58

Hampel\JobRunner\Cli\Command\RunJobs->execute() at /<path_to_forum>/src/vendor/symfony/console/Command/Command.php:255

Symfony\Component\Console\Command\Command->run() at /<path_to_forum>/src/vendor/symfony/console/Application.php:992

Symfony\Component\Console\Application->doRunCommand() at /<path_to_forum>/src/vendor/symfony/console/Application.php:255

Symfony\Component\Console\Application->doRun() at /<path_to_forum>/src/vendor/symfony/console/Application.php:148

Symfony\Component\Console\Application->run() at /<path_to_forum>/src/XF/Cli/Runner.php:109

XF\Cli\Runner->run() at /<path_to_forum>/cmd.php:15


hg:run-jobs [-t|--max-execution-time MAX-EXECUTION-TIME] [--manual-only]

Any ideas?

You're storing internal_data on AWS ??

Probably not a good idea to have the lock file written there, given how frequently it is updated - would be better on your local FS.

There isn't currently a configuration option to override the location of the lock file - I should probably add one.

What do you have codeCachePath and tempDataPath set to in the config?
 

VersoBit

Active member
You're storing internal_data on AWS ??

Probably not a good idea to have the lock file written there, given how frequently it is updated - would be better on your local FS.

There isn't currently a configuration option to override the location of the lock file - I should probably add one.

What do you have codeCachePath and tempDataPath set to in the config?
We have an extensive amount of traffic on the community this forum is serving, we have to run our servers across multiple nodes, easiest way to have our data accessible across the cluster in the DC was to use an S3 host.

We use Redis to its full extent and have our MySQL hosted externally as well.

codeCachePath =/<path_to_forum>/local_cache/code_cache
tempDataPath = /<path_to_forum>/local_cache/temp_data
 

Sim

Well-known member
Okay - I'll look at adding an option to allow you to specify where to write the lock file ... /<path_to_forum>/local_cache/run-jobs-lock.txt is probably a good place for it.

I assume you're not running the CLI Job Runner on multiple nodes - just one of them, right?
 

VersoBit

Active member
Okay - I'll look at adding an option to allow you to specify where to write the lock file ... /<path_to_forum>/local_cache/run-jobs-lock.txt is probably a good place for it.

I assume you're not running the CLI Job Runner on multiple nodes - just one of them, right?
Correct, we have the process running on the master node.
 
  • Like
Reactions: Sim

titocento

Member
hi, thanks for the plugin, great job. I have a problem with one forum tho, the usual error:

"There are scheduled jobs outstanding which have not run. Jobs may not be getting triggered when expected."

Weird thing is that executing command manually (without any parameter), it says all jobs have been executed, but a second after i do hg:show-jobs and i get this:

1605273925505.png

It seems it isn't registering cron last run, i have it set to run every minute (without params either). Any idea on how to fix that?

Regards,
 

Sim

Well-known member
That's strange - it is indicating that the Cron task has never been run.

Look in your xf_job table - there should be an entry for "cron" with the last_run_date column set - if it is blank, then your XF crons aren't running.

Temporarily turn off your unix cron task and then run hg:run-jobs -vvv to show full debug output and either post here or message me with the log output.

What version of XF are you running? Which version of the addon?
 
Top