XF 2.0 Cron job / run of code every less than 60 seconds and CURL

[Scandal]

I have this problem that needs solution / idea:

Well, we need to run a cron job every less than 60 seconds OR to run a code on every page load.
This code will check if there are some records on a database table and if yes will make a CURL request and then clear the table. If new records exists on next check, then again a CURL and clearing the table etc.

If we run it on every page load, of course on the most times no CURL will triggered but I have a suspicion that in case the CURL triggering, it will affect seriously on page load time (big delay to return the page to visitor).

What do you suggest for this case?

Basically, does the cron job system of XF2 running on the same page request a visitor do (= so again delay on page load cause the CURL) ?
I think the vBulletin software was using a "fake" image load, that basically triggers the cron job.

If XF2 do the same, I could extend the main class of cron core so my code will run by the cron side on every page load.

Let me know what you think.

 

[S Thomas]

I would rather rethink what you're trying to achieve. If you really need to load an external page on page load, I would shift that into client side and do that via AJAX.

This code will check if there are some records on a database table and if yes will make a CURL request and then clear the table. If new records exists on next check, then again a CURL and clearing the table etc.

This sounds like really not what anyone would ever want to do. So yea, I would rethink what you're trying to do.

 

[Sim]

The cron/job functionality in XF2 can handle this.

Have a cron function which calls \XF::app()->jobManager()->enqueueUnique() to trigger your job and to make sure that there is only ever one job running at a time.

Then just define your job which will do the database lookup and cURL request.

Or your cron code could do the database lookup and trigger a job to perform the cURL request only if the database conditions are met.

Plenty of examples in \XF\Cron\* and \XF\Job\*

 

[Scandal]

Thomas, it is very necessary for me to be inside XF2 app environment. If I make ajax request will means duplicate load of XF2 page for each visitor.
It is not just an external page load, it is an API call (with API Key etc and database checks which means php side is necessary).

@Sim I know how to add a new cron job via the admin control panel - I have done it with other addons I have develope - but as I know it allows me at least to run each minute.
But, maybe you suggest something else with that message?
Maybe I have to extend a specific XF2 class so the code runs one time on any page load (or at least every 5 - 15 seconds) and uniquely?

 

[Sim]

Why do you need it to run so frequently?

Even unix cron only runs once per minute. To run things more frequently, cron is the wrong task - and I wouldn't be relying on XF2's cron system either - you'll run the risk of breaking other important functions because they will never get a chance to run if your task is dominating the job queue.

Basically: if you're running the task every 5 seconds, and your task takes 4 or more seconds to run, then no other tasks will ever get a chance to run.

If you need something to run more frequently than once per minute, you're basically looking at needing to daemonise your process to allow it to run continuously and manage its own frequency internally.

I'd seriously suggest you revisit why you need it to run this way and see if there's a better architecture for it.

Perhaps if you could explain what it is you are trying to achieve in more detail, we might be able to help.

 

[S Thomas]

I'd seriously suggest you revisit why you need it to run this way and see if there's a better architecture for it.

Perhaps if you could explain what it is you are trying to achieve in more detail, we might be able to help.

^This.

 

[mikez006]

Scandal is working on an addon for me so let me explain the situation.

We use browser Push notifications (pushcrew.com) using their API. Members get push notifications when certain events happen and currently push notifications are sent every minute via cron job.

For one particular event, 60 seconds is too long. We want to lower that to around 10 seconds. There's no way to single out this one event, so we need to change the time for all push notifications.

This is the reason behind running the cron under 60 seconds.

 

[DragonByte Tech]

@Sim is absolutely correct, if you need crons running that frequently then the XF2 cron system is not the right way to go. You need a server-side cron that calls a PHP file. Doing this in the browser is going to cause no end of problems since you cannot rely on there being an active user.

Ignore how many active users your forum has on average, it's never a guarantee that you have anyone browsing the site at any given time.

XF2's crons should be considered "you know, it would be neat if you could run at this time, but if not, no worries, just sort it out when you get to it".

Server-side crons are more reliable because if your server is offline, you have bigger problems than your crons not running. The Linux subsystem also has more advanced process control so multiple crons can run simultaneously, eliminating the problem of a backlog.

* * * * */10 /usr/bin/php /home/username/public_html/cmd.php --quiet your-namespace:send-push-notifications

I think that's the right way to go about it...?


Fillip

 

[S Thomas]

This is a job for a daemon as @Sim has described.

 

[Scandal]

While Mike described to you what we need to achieve, an other idea I had is to send each push notification immediately after the event triggering.
For example: user A is starting a new conversation with user B who has enabled the push notifications -> via CURL + the pushcrew API will send the necessary notification (by extending the necessary class of XF2 and do the correct checks).

1. The only suspicion I have about it is that maybe cause delay to post for example the new conversation etc cause delay to the complete of the CURL execution.

2. edit: and an other idea is via JavaScript: after submit of something by a user (new reply / new conversation) to call on the background of the browser an AJAX call to the necessary php / controller to send the notifications.

What do you think?

 

[S Thomas]

1.) Do it after the post save event
2.) What I suggested.
3.) The correct approach is still a daemon so you can send out bulk notifications instead of just one each time so you don't flood your customers. You could implement a queue type into XF aswell, but well.

 

[Sim]

While Mike described to you what we need to achieve, an other idea I had is to send each push notification immediately after the event triggering.
For example: user A is starting a new conversation with user B who has enabled the push notifications -> via CURL + the pushcrew API will send the necessary notification (by extending the necessary class of XF2 and do the correct checks).

1. The only suspicion I have about it is that maybe cause delay to post for example the new conversation etc cause delay to the complete of the CURL execution.

2. edit: and an other idea is via JavaScript: after submit of something by a user (new reply / new conversation) to call on the background of the browser an AJAX call to the necessary php / controller to send the notifications.

What do you think?

I think in this specific case, sending the notification immediately is the correct course of action. Yes, there may be a slight delay - but it's only going to affect that one user and posting content doesn't have to be a high performance action.

The alternative would be to use queues.

When saving the conversation, push the notification event to a queue and then your queue worker (which is run as a long-running process, perhaps using Supervisor to keep it running) will pick up the event and send the notification. Queues are ideal for making high volume or potentially long-running tasks run independently of user processes.

You could use an external queue service using Beanstalkd, Amazon SQS or Redis and then build your queue worker using Laravel - https://laravel.com/docs/5.6/queues. You still need to make a call to the queue service from your XF2 app when saving the conversation, but you're only connecting to a local server (in Amazon SQS case, only if you're running on AWS), so the delay should be minimal.

Note that XF2's job system is itself basically a queue mechanism - but it has the limitation that it doesn't run as a daemonised process and so can't run in near-real-time like a long-running queue worker would.

 

[Scandal]

A very basic notice is this: only one push notification have to be sent each time, so, even queues system doesn't needed.
Examples:
User A starts a new conversation to B -> only B will receive the push notification (1 notification).
User A will reply to user's B thread -> only B will receive the push notification (= not all the subscribers of the thread).
The notification is one each time, so one API call.

For this reason I think that no cron / XF2 or Server needed finally. It means server load / lots of checks, with no really reason. If we make the API call immediately, we achieve too the zero delay for the notification to be sent.
What do you think based on this info?

 

[Sim]

I think that's the best way to start - keep it simple - try it and see if the users complain.

The delay users will see when submitting a conversation message will depend on the performance of the web service you are calling to send the notification. If it's only a couple of seconds maximum, I don't think users will mind.

That is, unless you're actually trying to replicate near-real-time chat via private messages, in which case, any delay is likely to be problematic.