XF 2.2 Do enqueueUnique jobs with the same uniqueId run concurrently?

[CStrategies]

We have a large database updating process that is run whenever a user takes certain actions. It was causing deadlocks when the site had a lot of activity, so we converted it to a job with a uniqueId, passed certain params that were unique to that user, and made sure manual was set to "false" so the job would run in the background.

There is some evidence that would suggest that when traffic is high, some of these jobs aren't actually being run, or maybe they're being dropped. It's difficult to diagnose without really understanding how the job system works, which seems to be thinly explained in the developer docs.

Code:

$jobManager->enqueueUnique('updateJobUniqueId', 'MyName\MyAddon\Job\MyJobClass', [
                'thisUserData' => $thisUserData
            ], false);

The way I think it works is that the job manager will see if there is already an updateJobUniqueId job running, and if so, it will queue this next instance of the job to start when the current one ends.

What I hope is not happening but I suspect may be the case: the job manager sees updateJobUniqueId is already running, so it does not run the new job at all.

I thought about not using enqueueUnique but then I'm back to the deadlock issue, defeating the purpose.

 

[Chris D]

Unfortunately a unique job is supposed to be unique in that there should only be one instance of that job queued at any one time. Enquiring a new unique job with the same unique key will overwrite the existing one.

 

[Kirby]

What I hope is not happening but I suspect may be the case: the job manager sees updateJobUniqueId is already running, so it does not run the new job at all.

Almost - the last call wins.

A unique job is ... well.. unique.

There can't be more than one unique job with the same ID queued at any time.

Take a look at XF\Job\Manager to better understand how the job system works.

 

[digitalpoint]

In case it's not obvious, you can run the same job class for different unique IDs. A common approach would be to hash the part of the job that's actually unique (in the example above, that would be the user data). That way if you queue the same job with also the same data, it would only run once.

Something along the lines of:

$jobManager->enqueueUnique('mySweetJob_' . md5(json_encode($thisUserData)), 'MyName\MyAddon\Job\MyJobClass', [
    'thisUserData' => $thisUserData
], false);

If there's nothing unique about the data being passed to the job or you want to run the same task multiple times for whatever reason, you could generate a unique ID that has a random string appended to it. Something like 'mySweetJob_' . \XF::generateRandomString(16).

 

[CStrategies]

After examining the jobManager code, I still have one dumb question:

Despite the term being "enqueue", do jobs run simultaneously or consecutively if they have different IDs?

 

[Kirby]

IIRC jobs might run simultaneously (depends on job triggers, execution time, etc.)