XF 2.3 Cron Job

[Lee]

I'm not sure if I have approached this correctly, or if anybody can think of a better, more efficient way to do this.

I have a cron that selects x amount of threads based on the total amount of replies.

If the replies are greater than an option defined number, then I add them to a table to be processed.

These threads are then then sent to a third party API to be evaluated. At the moment there is one API call per thread. I have used the job system to "Enqueue" the x amount of tasks. Once they have been processed, they are marked as such so they aren't selected for processing again.

This works very well, but takes up lots of API calls / money - and also means for existing threads that need processing, it can be a VERY lengthy process.

Is there a better, more efficient way to approach this?

 

[Lee]

Bump.

 

[Jeremy P]

It's not super clear to me what you're asking. Your cron can queue up a job (or series of jobs), which are designed to handle long-running tasks outside of the usual page view request/response cycle, and a single job can be broken into multiple steps or batches which pause and resume over time. That sounds like what you have going already.

If you're asking if you can batch multiple threads into a single API call (and how you might go about that), that's a question for the people operating the third-party API.

 

[Sim]

I have used the job system to "Enqueue" the x amount of tasks. Once they have been processed, they are marked as such so they aren't selected for processing again.

If you're creating individual Jobs for each of the threads, that's not the most efficient way.

Take a look at some of the Jobs used by the XF core - some of them use restartable logic where each time the job starts it does a query to select a bunch of items to be processed, then processes them sequentially until the time limit is up. Then if it hasn't finished processing all threads, it will automatically create a new Job to run in the future and resume processing. This will work because you are marking the threads as processed as you go, so the new query will only select those threads not yet processed.

Jeremy's comment about batching the API calls would be the only way to save on costs - it would depend on whether the external API allows that and whether it actually reduces cost to do so.