GoodForNothing Shell Defer 1.0.4

[Mr. Goodie2Shoes]

Mr. Goodie2Shoes submitted a new resource:

GoodForNothing Shell Defer - Move XenForo's defer task to shell!

I actually made it a long time ago for a friend who wanted to make sure the cron entries run properly even without having any visitors for a day.

The script will be executed in a loop until all the deferred process is completed. And a 'defer lock' will be created to make sure that only one instance of the defer process is running.

And after seeing this thread, I wanted to post it to the public.

To make the add-on work, all you have to do is upload the files from the upload directory....

Read more about this resource...

 

[dono]

do we need to remove the deferred trigger from the PAGE_CONTAINER template ?

 

[Mr. Goodie2Shoes]

do we need to remove the deferred trigger from the PAGE_CONTAINER template ?

Totally optional, if you have too many visitors you probably want to remove it

 

[dono]

please define `too many' in your opinion.. 600 visitors is too many ? and visitors in here including guests and search engines ?

 

[Mr. Goodie2Shoes]

hmmm... you'll save some resource if you remove the trigger from PAGE_CONTAINER

 

[Xon]

hmmm... you'll save some resource if you remove the trigger from PAGE_CONTAINER

Your addon already contains a template modification which does that

 

[Mr. Goodie2Shoes]

Your addon already contains a template modification which does that

I've mentioned in the description that importing the XML is optional to remove the public call of deferred.php and @dono wanted to clear it up I guess

 

[Mr. Goodie2Shoes]

Mr. Goodie2Shoes updated GoodForNothing Shell Defer with a new update entry:

Fix for potential server hogging.

Fix for potential server hogging.

Read the rest of this update entry...

 

[Xon]

@Mr. Goodie2Shoes I dug into why the performance impact was happening.

What was happening is a cron deferred task was coming up, and then instantly requeuing itself for now + some seconds. Except 'now' is cached at the start of the script. No matter how long the defer script runs for.

IMO; After the sleep(10), it would be a good idea to add:

XenForo_Application::$time = time();

 

[Mr. Goodie2Shoes]

Mr. Goodie2Shoes updated GoodForNothing Shell Defer with a new update entry:

Update on previous fix.

This update fixes the same issue, as @Xon pointed out, there was a problem with fetching the deferred entries because of XenForo_Application::$time being cached.

Read the rest of this update entry...

 

[Mr. Goodie2Shoes]

IMO; After the sleep(10), it would be a good idea to add:

If this was the issue (which I think it is), I think it's safe to remove sleep() from the script...

 

[Xon]

@Mr. Goodie2Shoes

Rather than use a lock file, how about using the database to ensure only one instance is running at a time:

$db = XenForo_Application::getDb();

$lock_result = $db->fetchOne("select get_lock(?, ?)", array('cli-defer-lock', 1));
if ($lock_result !== 1)
{
  exit(0);
}

This has the advantage of automatically cleaning up if the process goes away, and it works across multiple worker nodes.

 

[Mr. Goodie2Shoes]

ah... learned something new today

 

[Mr. Goodie2Shoes]

Mr. Goodie2Shoes updated GoodForNothing Shell Defer with a new update entry:

Moved 'defer lock' to database.

As suggested by Xon, I have moved the defer lock to database which, hopefully, will provide some improvements

Just uploading and over-writing the files inside upload/cli folder will suffice.

Read the rest of this update entry...

 

[HWS]

Why do you need a "defer lock"?

If you don't start defer with the shell but leave it in the PAGE_CONTAINER it can also be triggered more than once at the same time.

 

[Xon]

Why do you need a "defer lock"?

If you don't start defer with the shell but leave it in the PAGE_CONTAINER it can also be triggered more than once at the same time.

How the deferred task system is designed and works is quite basic. It really doesn't scale well without a lot of superfluous DB reads & writes (due to speculative fetches+ discards, and then lock contention on discarding).

And if the deferred tasks themselves aren't designed properly, they are prone to deadlock and the task being executed many times at once.

 

[HWS]

How the deferred task system is designed and works is quite basic. It really doesn't scale well without a lot of superfluous DB reads & writes (due to speculative fetches+ discards, and then lock contention on discarding).

Our forum is not very small and we never had problems with defer (also manually triggered from the shell because it would run too often from PAGE_CONTAINER).

@Mike : What do you think about it? Does defer need a lock?

 

[Mr. Goodie2Shoes]

Well my intention is purely something like:

There's already one deferred process taking place and it's trying its best to finish the task ASAP. No need to waste server resources with additional instances.

I might be wrong though as concurrent process means the task timing will be reduced. Oh well...

 

[Mike]

@Mike : What do you think about it? Does defer need a lock?

The deferred job runner is atomic within a particular job. The process that successfully updates the job is the one that runs it; others that have tried to grab that job in the interim will have their update fail so they will skip the job. So in that regard, another lock isn't explicitly needed. Multiple simultaneous runners shouldn't be an issue unless it starts overloading the server (if you have multiple cores available then multiple jobs could run simultaneously with minimal slowdown).

 

[Mr. Goodie2Shoes]

The deferred job runner is atomic within a particular job. The process that successfully updates the job is the one that runs it; others that have tried to grab that job in the interim will have their update fail so they will skip the job. So in that regard, another lock isn't explicitly needed. Multiple simultaneous runners shouldn't be an issue unless it starts overloading the server (if you have multiple cores available then multiple jobs could run simultaneously with minimal slowdown).

So that means it wont hurt to add an explicit lock just to prevent any more unnecessary executions?