Lack of interest e-mail log

[Alpha1]

It would be useful to have an email log so that we can see exactly what emails xenforo has sent.
I'm actually surprised that this doesn't exist. Especially since there is an email bounce log, but no email log.

Example of why its useful: I've just sent out a mail shoot, but this somehow was interupted. Now I need to resend starting at a certain user ID. But to which users did xenforo send? There seems to be no way to find out.

 

[Chris D]

A better solution is probably resumable email sending, which we do have in XF2 already. It runs as a manual job - like cache rebuilds etc. and so if that is ever interrupted, it can be resumed from where it left off.

 

[Alpha1]

Thats very true.

An even better solution is to have a function to send all email at once. Then we would not need to pick up where it went wrong. I need to wait multiple days for the uberslow process to complete. It went faster in 2005 on vbulletin. 13 years later its not workable.

 

[Alpha1]

A better solution is probably resumable email sending, which we do have in XF2 already. It runs as a manual job - like cache rebuilds etc. and so if that is ever interrupted, it can be resumed from where it left off.

What strikes me in XF1 is that if a hiccup occurs, I get the message:

Your email was sent to 199,234 users.

Yet, this did not happen at all. Its then a matter of trying to find out what the last userid was. But the process doesnt display the userID. Is there any way to find it?

 

[HWS]

Your email was sent to 199,234 users.

If I would need to send mail to almost 200k receivers, I would certainly not use Xenforo for it.

 

[Alpha1]

If I would need to send mail to almost 200k receivers, I would certainly not use Xenforo for it.

IMHO Xenforo should be able to do this well. But it is not equipped for it.
Any method of sending should feed back into the xenforo bounced mail, so that accounts are properly deactivated.

 

[Kirby]

If I would need to send mail to almost 200k receivers, I would certainly not use Xenforo for it.

We've been sending Newsletters with vBulletin to approx. 800K subscribers on a single forum several years ago, so why shouldn't XenForo be able to handle 200K?

 

[Sim]

Yet, this did not happen at all.

What ESP are you using to send the emails? Or are you sending via your own SMTP server - or via PHP directly?

Was the problem at the XenForo end or at the ESP end?

 

[Alpha1]

I was using my mail server (postfix)
It may be a server hiccup that caused not all mails to be sent. For example a disruption of communication between virtual instances. Its too bad that XF1 then skips right to the 'Your email was sent to X users' message. Even if that did not happen.

 

[ibnesayeed]

Here is a related suggestion, but not exactly the same. https://xenforo.com/community/threads/add-sent-email-count-in-the-acp-statistics-plots.132539/

 

[Sim]

I'm pretty sure that XF1 simply ran through a tight loop trying to do all the sends at once - so if your mail sending system was not able to handle a large volume of sending, or if you ran out of memory or anything, it would fail and was not resumable. From what I understand of the code, it also has a hard limit of 2 minutes to complete the task or it will fail.

At the end of the day, XF1 was not really suitable for sending email to a large number of people unless you had extremely robust infrastructure - and even then I can forsee timeouts occurring.

Is this the first time you've sent emails to a large number of people from XF1?

Trying to send hundreds of thousands of emails all at once is a bad idea - you'd need to batch them up and send them in a controlled manner. An external newsletter service would be a much better choice IMO.

The XF1 addon I built for myself to send a weekly digest email to users used the mail queue system rather than trying to send all the emails at once. I also use an ESP for more robust sending.

XF2 has a much more robust system, although I've not tested it with large numbers of recipients yet.

 

[Alpha1]

Is this the first time you've sent emails to a large number of people from XF1?

Yes. Though XF is sending thousands of transactional email per week.

Trying to send hundreds of thousands of emails all at once is a bad idea - you'd need to batch them up and send them in a controlled manner.

There is queuing on the mail server.

An external newsletter service would be a much better choice IMO.

I have not seen any solid addons that give access to cheap providers and feeds bounced mail back into XF.

The XF1 addon I built for myself to send a weekly digest email to users used the mail queue system rather than trying to send all the emails at once.

Does the addon digest forum only or addons as well?

 

[Sim]

There is queuing on the mail server.

The problem is that if it takes more than 2 minutes for your mail server to accept all of the emails (which is quite likely for such a large volume), then the send will time out (without error ).

My guess is that your mail server wasn't able to process the emails fast enough - SMTP is quite a chatty protocol, so I'm assuming the throughput would be relatively low.

That's one of the reasons I implemented API level mail sending via an ESP - they advised that throughput would be much higher than using SMTP to send.

That being said, it would still take some time to send such a large batch - my XF1 server has been sending bulk emails at a rate of around 5,000 emails per hour via SparkPost - which is quite slow. Their documentation states that throughput can be optimised/increased by sending multiple batches in parallel which I don't do.

Back to the original problem - it would be interesting to see what your mail server logs indicate - what time did the server start receiving the emails and what time did it stop? (Also - how many emails were actually processed?) If it is more than 2 minutes, then that's likely your problem - XF1 timed out before your server accepted all of the emails.

Batching up emails in XenForo (not your mail server) would avoid this issue - which is what XF2 does. But of course, it will still take some time to send all of the emails.

I have not seen any solid addons that give access to cheap providers and feeds bounced mail back into XF.

Yeah, the bounced mail back into XF is the trick.

I like SparkPost as an ESP, but they deliver mail bounce notifications via webhook rather than as returned email, which requires quite a bit of programming to handle. I've built this processing for my own sites, but it relies on an external (Laravel) application to process the bounce notifications and pass them on to XenForo - so it's not really something I'm going to release at this point.

Also, your costs will be quite high for such high volumes of emails.

Once I get my sites upgraded to XF2 I might spend some time investigating the mail throughput issue and see if I can come up with a solution for implementing high volume mail sending - ie batching the emails and sending them in parallel using concurrent sending connections.

 

[HWS]

Does the addon digest forum only or addons as well?

There is no need for an add-on. Our XF1 forum sends all mail over Amazon SES very well and registers bounces without any problems out of the box.