• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

XF 1.5 Bypass E-mail Defer Queue

nrep

Well-known member
#1
We send out a newsletter using the XF bulk mail system, however since upgrading the XF 1.5 it has become painfully slow. From some helpful advice on the forums, I found out this was because XF 1.5 handles emails by adding them to a defer queue.

Is there a way to bypass this? On XF 1.4 we can send 5k e-mails in less than a minute. On XF 1.5 it takes closer to an hour.
 

eva2000

Well-known member
#2
Indeed my first admin email to user on upgraded XF 1.5 was definitely much slower than when on XF 1.4 ~ 11-13 emails/sec !
 

Chris D

XenForo developer
Staff member
#3
You're right, it is slower.

But also it's more consistent and reliable.

One reason for it changing was the not uncommon occurrence of encountering some sort of interruption part way through the process. The process often wasn't resumable so you'd either need to start again or accept that it wasn't finished. Now it runs on the Deferred system if it is interrupted, it can be resumed once the interruption is over.
 

eva2000

Well-known member
#4
You're right, it is slower.

But also it's more consistent and reliable.

One reason for it changing was the not uncommon occurrence of encountering some sort of interruption part way through the process. The process often wasn't resumable so you'd either need to start again or accept that it wasn't finished. Now it runs on the Deferred system if it is interrupted, it can be resumed once the interruption is over.
but i assume resumption wouldn't occur if php timed out and crashed while processing the slow email queue ?
 

nrep

Well-known member
#5
I understand why it's like that, but I'd happily have the option to sacrifice that for mass e-mails. It's taking hours as opposed to minutes to send an e-mail, which is quite frustrating.

Is there a way to disable this (or a hint on how to code something)? Any tips would be most appreciated :).
 

Chris D

XenForo developer
Staff member
#6
but i assume resumption wouldn't occur if php timed out and crashed while processing the slow email queue ?
It's a manual deferred task, so yeah it would. It's done in batches of X emails, and has logic in there to cut that short should it take longer than Y. The upshot is, each time it is finished a batch, the current position is saved back to the xf_deferred table. If all hell broke loose (short of losing the contents of the xf_deferred table), there's enough information logged in there to allow it to continue (roughly) where it left off. You'd have a banner on the Admin CP home page stating "There are manual rebuild processes that have been stopped before completion. Click here to complete them."

I understand why it's like that, but I'd happily have the option to sacrifice that for mass e-mails. It's taking hours as opposed to minutes to send an e-mail, which is quite frustrating.

Is there a way to disable this (or a hint on how to code something)? Any tips would be most appreciated :).
You originally said it's taking "an hour" rather than hours. Honestly, an hour doesn't seem that bad to me. You don't have to sit there and watch it. Open another tab, carry on with your life, and you can be confident it's going to finish the job. You could even close your browser and finish it later if you had to.

There's no way to disable it.
 

nrep

Well-known member
#7
You originally said it's taking "an hour" rather than hours. Honestly, an hour doesn't seem that bad to me. You don't have to sit there and watch it. Open another tab, carry on with your life, and you can be confident it's going to finish the job. You could even close your browser and finish it later if you had to.

There's no way to disable it.
The hour was just a comparison for the site that does 5k e-mails, I've sent out newsletters for 7 sites so far today (and more to go).

Some of these sites have 20k subscribers, so I'm sure you can understand why this is frustrating for me and worth seeing if there is a way around it.
 

Chris D

XenForo developer
Staff member
#8
No. I don't understand why it is frustrating. This is what tabbed browsing was invented for.

Ultimately, the benefits of this approach far outweigh the time it takes. There's no configurable option to change it, and it's not a trivial thing to change it back.

It would therefore require custom development, but I'd still urge against it for the reasons already mentioned.
 

nrep

Well-known member
#9
Fair enough, well I find it frustrating. If anyone else has any suggestions on how I may be able to bypass this, I'd be interested to know.
 

eva2000

Well-known member
#10
It's a manual deferred task, so yeah it would. It's done in batches of X emails, and has logic in there to cut that short should it take longer than Y. The upshot is, each time it is finished a batch, the current position is saved back to the xf_deferred table. If all hell broke loose (short of losing the contents of the xf_deferred table), there's enough information logged in there to allow it to continue (roughly) where it left off. You'd have a banner on the Admin CP home page stating "There are manual rebuild processes that have been stopped before completion. Click here to complete them."
yeah PHP timeouts are the only issue i encounter now on upgraded XF 1.5, so will need to adjust my php time outs to account for this on the server level
 

Floyd R Turbo

Well-known member
#11
I'm hitting this issue today. Prior to 1.5.x, when I sent out a bulk mail (to only about 700-900 users) it flew, taking maybe 5 or max 10 minutes, 10-15 at a time.

Today, I'm sending one to ~900 users and it is literally sending them one at a time. After about 5 minutes, it's on #45.

What's going on here?

To be fair, a few things have changed since my last bulk mailer:

Changed from GoDaddy VPS to a non-GoDaddy VPS
Changed from using Pepipost for forum email to server-based
Changed from php5.6 to php 7.0

I think that's about it though...is the quantity of messages sent in each batch driven by an ACP setting? I don't think I've ever seen one...
 

Floyd R Turbo

Well-known member
#13
Thanks @Brogan...I've never changed this value though, so what could be the explanation for it flying along before, but now going to a crawl? It's on 202 now...30 minutes into the queue
 

Floyd R Turbo

Well-known member
#15
What specifically? Before I used Pepipost, when still on the GoDaddy VPS, I had sent out bulk mailers and it went very fast (using the server-based email)
 

Floyd R Turbo

Well-known member
#16
Addtionally, the rebuild time has always seemed to be about the same, 8 seconds (I haven't added any other override into the config.php file, ever)

So before, it would send out 10-15 emails in each rebuild, now it sends only one at a time.
 

Floyd R Turbo

Well-known member
#18
Ok, thanks. I don't think it's the mail transport method, I've used both -f parameter and SMTP before without issue. I'm guessing it's the php version and a related server setting. I know this is beyond the scope of XF but if anyone could point me to the server setting that would be much appreciated (have full root/WHM)