XF 1.5 Bypass E-mail Defer Queue

[nrep]

We send out a newsletter using the XF bulk mail system, however since upgrading the XF 1.5 it has become painfully slow. From some helpful advice on the forums, I found out this was because XF 1.5 handles emails by adding them to a defer queue.

Is there a way to bypass this? On XF 1.4 we can send 5k e-mails in less than a minute. On XF 1.5 it takes closer to an hour.

 

[eva2000]

Indeed my first admin email to user on upgraded XF 1.5 was definitely much slower than when on XF 1.4 ~ 11-13 emails/sec !

 

[Chris D]

You're right, it is slower.

But also it's more consistent and reliable.

One reason for it changing was the not uncommon occurrence of encountering some sort of interruption part way through the process. The process often wasn't resumable so you'd either need to start again or accept that it wasn't finished. Now it runs on the Deferred system if it is interrupted, it can be resumed once the interruption is over.

 

[eva2000]

You're right, it is slower.

But also it's more consistent and reliable.

One reason for it changing was the not uncommon occurrence of encountering some sort of interruption part way through the process. The process often wasn't resumable so you'd either need to start again or accept that it wasn't finished. Now it runs on the Deferred system if it is interrupted, it can be resumed once the interruption is over.

but i assume resumption wouldn't occur if php timed out and crashed while processing the slow email queue ?

 

[nrep]

I understand why it's like that, but I'd happily have the option to sacrifice that for mass e-mails. It's taking hours as opposed to minutes to send an e-mail, which is quite frustrating.

Is there a way to disable this (or a hint on how to code something)? Any tips would be most appreciated .

 

[Chris D]

but i assume resumption wouldn't occur if php timed out and crashed while processing the slow email queue ?

It's a manual deferred task, so yeah it would. It's done in batches of X emails, and has logic in there to cut that short should it take longer than Y. The upshot is, each time it is finished a batch, the current position is saved back to the xf_deferred table. If all hell broke loose (short of losing the contents of the xf_deferred table), there's enough information logged in there to allow it to continue (roughly) where it left off. You'd have a banner on the Admin CP home page stating "There are manual rebuild processes that have been stopped before completion. Click here to complete them."

I understand why it's like that, but I'd happily have the option to sacrifice that for mass e-mails. It's taking hours as opposed to minutes to send an e-mail, which is quite frustrating.

Is there a way to disable this (or a hint on how to code something)? Any tips would be most appreciated .

You originally said it's taking "an hour" rather than hours. Honestly, an hour doesn't seem that bad to me. You don't have to sit there and watch it. Open another tab, carry on with your life, and you can be confident it's going to finish the job. You could even close your browser and finish it later if you had to.

There's no way to disable it.

 

[nrep]

You originally said it's taking "an hour" rather than hours. Honestly, an hour doesn't seem that bad to me. You don't have to sit there and watch it. Open another tab, carry on with your life, and you can be confident it's going to finish the job. You could even close your browser and finish it later if you had to.

There's no way to disable it.

The hour was just a comparison for the site that does 5k e-mails, I've sent out newsletters for 7 sites so far today (and more to go).

Some of these sites have 20k subscribers, so I'm sure you can understand why this is frustrating for me and worth seeing if there is a way around it.

 

[Chris D]

No. I don't understand why it is frustrating. This is what tabbed browsing was invented for.

Ultimately, the benefits of this approach far outweigh the time it takes. There's no configurable option to change it, and it's not a trivial thing to change it back.

It would therefore require custom development, but I'd still urge against it for the reasons already mentioned.

 

[nrep]

Fair enough, well I find it frustrating. If anyone else has any suggestions on how I may be able to bypass this, I'd be interested to know.

 

[eva2000]

It's a manual deferred task, so yeah it would. It's done in batches of X emails, and has logic in there to cut that short should it take longer than Y. The upshot is, each time it is finished a batch, the current position is saved back to the xf_deferred table. If all hell broke loose (short of losing the contents of the xf_deferred table), there's enough information logged in there to allow it to continue (roughly) where it left off. You'd have a banner on the Admin CP home page stating "There are manual rebuild processes that have been stopped before completion. Click here to complete them."

yeah PHP timeouts are the only issue i encounter now on upgraded XF 1.5, so will need to adjust my php time outs to account for this on the server level

 

[Floyd R Turbo]

I'm hitting this issue today. Prior to 1.5.x, when I sent out a bulk mail (to only about 700-900 users) it flew, taking maybe 5 or max 10 minutes, 10-15 at a time.

Today, I'm sending one to ~900 users and it is literally sending them one at a time. After about 5 minutes, it's on #45.

What's going on here?

To be fair, a few things have changed since my last bulk mailer:

Changed from GoDaddy VPS to a non-GoDaddy VPS
Changed from using Pepipost for forum email to server-based
Changed from php5.6 to php 7.0

I think that's about it though...is the quantity of messages sent in each batch driven by an ACP setting? I don't think I've ever seen one...

 

[Paul B]

is the quantity of messages sent in each batch driven by an ACP setting?

It's based on the rebuild max value, which defaults to 8 (seconds): https://xenforo.com/help/config-php-options/

 

[Floyd R Turbo]

Thanks @Brogan...I've never changed this value though, so what could be the explanation for it flying along before, but now going to a crawl? It's on 202 now...30 minutes into the queue

 

[Paul B]

I think you answered that question yourself in your first post.

 

[Floyd R Turbo]

What specifically? Before I used Pepipost, when still on the GoDaddy VPS, I had sent out bulk mailers and it went very fast (using the server-based email)

 

[Floyd R Turbo]

Addtionally, the rebuild time has always seemed to be about the same, 8 seconds (I haven't added any other override into the config.php file, ever)

So before, it would send out 10-15 emails in each rebuild, now it sends only one at a time.

 

[Paul B]

You have changed server, mail transport, and PHP version.

It could be any one/combination of those things.

 

[Floyd R Turbo]

Ok, thanks. I don't think it's the mail transport method, I've used both -f parameter and SMTP before without issue. I'm guessing it's the php version and a related server setting. I know this is beyond the scope of XF but if anyone could point me to the server setting that would be much appreciated (have full root/WHM)

 

[Floyd R Turbo]

I'm still experiencing this issue. Can anyone point me in the right direction at to what setting, either in XF or in WHM/cPanel, would be the root cause of this issue?

When I initiate a bulk email, it just goes one at a time. So from reading this thread

yeah PHP timeouts are the only issue i encounter now on upgraded XF 1.5, so will need to adjust my php time outs to account for this on the server level

is it possible that the php timeouts are the issue? But where would this setting be, specifically? The only thing I can find with my limited knowledge is default_socket_timeout = 60 (seconds) so that doesn't seem likely.

It's a manual deferred task, so yeah it would. It's done in batches of X emails, and has logic in there to cut that short should it take longer than Y.

It seems like this could explain it, what seems to happen is that it initiates the first message and that seems to take a while - a lot longer than the one-at-a-time taking ~8 seconds (XF default max rebuild value). Then, it starts rolling along at one email every rebuild / 8 seconds, and stays that way.

I'm running php7 with php-fpm enabled, if that matters.

 

[Chris D]

Nothing I can recommend to change in XF. It should just work.