Sending large number of batch emails

[mjda]

On my forum I periodically need to send a large number of members an email. When this happens I usually get blocked from gmail and yahoo because I'm sending too many, too frequently. I'm wondering what is the best way to get around this? How to large forums send large numbers of emails at a time without getting blocked? For example, I'm sure when XF posts in the HYS there are probably a thousand, or more, emails sent to members watching that forum. I'm assuming these go in a queue of some sort?

That said, I'd also like the main emails (registrations, watching email notifications, etc) to not be queued up so they take forever to get to their destination.

Is this something that's possible with XF's setup or will I need an add-on for this?

Any info I can get on this would be much appreciated.

 

[GW2]

Have you considered using Amazon SES? It works with XF.

 

[TPerry]

I'll second SES... or a similar service. With Amazon, the cost is not that much. The only downfall is sometimes their sending IP is in a spam list or two, but so far it has not affected my deliverability. You can also (if you want to pay more) get a dedicated IP from them that does not have that issue.
I did (with SES) find that the add-on available in the RM was somewhat problematic for reporting on bounced emails during my testing, but that's easy enough to resolve with setting up separate emails for the bounced address. During testing doing this worked fine. This email test immediately returned as a hard bounce... similar emails using that add-on only returned soft-bounce if I was lucky to get a return at all.



You could probably set up a simple free gmail account for the bounce & unsubscribe addresses and check them via the setup.

 

[Suzanne O]

Look at your host's email settings in it's Cpanel.

 

[GW2]

I'll second SES... or a similar service. With Amazon, the cost is not that much. The only downfall is sometimes their sending IP is in a spam list or two, but so far it has not affected my deliverability. You can also (if you want to pay more) get a dedicated IP from them that does not have that issue.
I did (with SES) find that the add-on available in the RM was somewhat problematic for reporting on bounced emails during my testing, but that's easy enough to resolve with setting up separate emails for the bounced address. During testing doing this worked fine. This email test immediately returned as a hard bounce... similar emails using that add-on only returned soft-bounce if I was lucky to get a return at all.

View attachment 290180

You could probably set up a simple free gmail account for the bounce & unsubscribe addresses and check them via the setup.

Yes, that's basically what I do. I do not use gmail. However you could use any email provider. The cost of SES is minimal. The most significant benefit to me by SES is that my emails are 99.9% not blocked by Yahoo, gmail, or other email providers because of suspected spam.

As @Tracy Perry says above, I receive an occasional bounce using SES because of spam lists, but nothing significant.

And, I can send 50,000± emails per day at the rate of 14 per second.

 

[Suzanne O]

Yes, that's basically what I do. I do not use gmail. However you could use any email provider. The cost of SES is minimal. The most significant benefit to me by SES is that my emails are 99.9% not blocked by Yahoo, gmail, or other email providers because of suspected spam.

As @Tracy Perry says above, I receive an occasional bounce using SES because of spam lists, but nothing significant.

And, I can send 50,000± emails per day at the rate of 14 per second.

My own Cpanel does this too.

Email Accounts | cPanel & WHM Documentation

This interface lets you add, manage, and delete email accounts.

docs.cpanel.net

It's worth looking at having something like Cpanel.

I've been with hosts that don't have their own email system and they're shocking.

 

[mjda]

Thank y'all for the recommendation. Definitely something I'll look in to. Definitely gotta figure something out here soon.

And, I can send 50,000± emails per day at the rate of 14 per second.

What manages this rate? For example, what if I send over 1,000 at once? Will that push over the 14 per second limit or will they all queue up and send at that rate?

 

[TPerry]

What manages this rate? For example, what if I send over 1,000 at once? Will that push over the 14 per second limit or will they all queue up and send at that rate?

It's 14 messages a second. I doubt even sending 1000 at once will exceed that limit. You are looking at roughly 840 emails a minute.
It is going to be rather hard for you to send 14 emails a second generally.
If you do find an issue.. you can usually request an increase if you can justify it. Just for grins and giggles... mine is at 14... I just requested an increase to 25... so I'll see how they respond..... and they just repsonded.. I'm increased to 25 per second... and it was done within 2 minutes of requesting it.

 

[mjda]

It's 14 messages a second. I doubt even sending 1000 at once will exceed that limit. You are looking at roughly 840 emails a minute.
It is going to be rather hard for you to send 14 emails a second generally.

I was just thinking about when I send emails via the admin panel, it will "send" 1,000+ emails in just a few seconds. Without some kind of queue that would exceed the limit, unless the limit is based on an average of some sort.

 

[TPerry]

I was just thinking about when I send emails via the admin panel, it will "send" 1,000+ emails in just a few seconds. Without some kind of queue that would exceed the limit, unless the limit is based on an average of some sort.

Yeah... XF is lacking in some aspects in that. There should be a batch throttle for sending direct from the ACP instead of dumping them all at once.
I think (but am not sure) that SES will queue those messages and send them at the throttled limit.. but I don't send enough to have ran into it and honestly have not taken the time to research it since it doesn't affect me currently and I have other concerns I am tied up pursuing.

 

[mjda]

There should be a batch throttle for sending direct from the ACP instead of dumping them all at once.

Yes, this! If XF had this, then I wouldn't need to be looking at these other options.

 

[eva2000]

What manages this rate? For example, what if I send over 1,000 at once? Will that push over the 14 per second limit or will they all queue up and send at that rate?

Like @Tracy Perry suggested, you can increase the limits but read the AWS SES guidelines on when to increase them as SES can also automatically increase the rates based on your usage https://docs.aws.amazon.com/ses/latest/dg/manage-sending-quotas-request-increase.html

Automatically increased sending quotas​

When your account is out of the sandbox and you're sending high-quality production email, we might automatically increase the sending quotas for your account. Often, we automatically increase these quotas before you actually need them to be increased.

To qualify for automatic rate increases, all of the following statements have to be true:

• You send high-quality content that your recipients want to receive –Send content that recipients want and expect. Stop sending email to customers who don't open your email.
• You send actual production content – Sending test messages to fake email addresses can have a negative effect on your bounce and complaint rates. Also, sending messages only to internal recipients makes it difficult to determine if you're sending content that customers want to receive. However, when you send your production messages to non-internal recipients, we can accurately assess your email-sending practices.
• You send near your current quota – To qualify for an automatic quota increase, your daily email volume should regularly approach the daily maximum for your account without exceeding it.
• You have low bounce and complaint rates – Minimize the number of bounces and complaints that you receive. Having a high number of bounces and complaints can have a negative impact on your sending quotas.

SES now is more strict for SES approvals and rate limit manual increases though - probably because they can let auto limit increase policy handle legit needs

I'm grateful to have the SES limits I have at a time when it was much easier to get approvals

 

[Suzanne O]

Like @Tracy Perry suggested, you can increase the limits but read the AWS SES guidelines on when to increase them as SES can also automatically increase the rates based on your usage https://docs.aws.amazon.com/ses/latest/dg/manage-sending-quotas-request-increase.html



SES now is more strict for SES approvals and rate limit manual increases though - probably because they can let auto limit increase policy handle legit needs

I'm grateful to have the SES limits I have at a time when it was much easier to get approvals

View attachment 290223

The only pitfall with AWS products is they can rip you right off.
Would be much better to let their customers know of the price list first.

 

[eva2000]

The only pitfall with AWS products is they can rip you right off.
Would be much better to let their customers know of the price list first.

For many AWS products yes, but SES pricing is pretty transparent as it's usage based.

 

[Miri]

I was running a website on a well-known CMS and it had a built-in "component" that gave me complete freedom to send and schedule queue for all emails and notifications, even considering the restrictions of my host. This helped to avoid being blacklisted from email services or even blocked by my host.

With XenForo, even when dealing with a small forum, I sometimes encounter difficulties with email. I have not tried external services and wonder why there is no noteworthy add-on available for email management. It would be great to manage everything internally with an add-on, allowing you to set up intervals and scheduled deliveries based on your specific needs.

 

[mjda]

I was running a website on a well-known CMS and it had a built-in "component" that gave me complete freedom to send and schedule queue for all emails and notifications, even considering the restrictions of my host. This helped to avoid being blacklisted from email services or even blocked by my host.

With XenForo, even when dealing with a small forum, I sometimes encounter difficulties with email. I have not tried external services and wonder why there is no noteworthy add-on available for email management. It would be great to manage everything internally with an add-on, allowing you to set up intervals and scheduled deliveries based on your specific needs.

Yes, I agree 100%. Being able to handle a mail queue from within XF's core would be the way to go, at least for me. I have no issues other than when I send batch emails to a specific group of users on my forum. That happens a few times a year and every time it happens, I get returned emails because I'm sending too many, too soon. Having to sign up, and pay, for another service just for a few times a year isn't ideal. However, at this point I really have no choice because I can't keep getting blocked and having transactional emails go undelivered.

 

[GW2]

Thank y'all for the recommendation. Definitely something I'll look in to. Definitely gotta figure something out here soon.



What manages this rate? For example, what if I send over 1,000 at once? Will that push over the 14 per second limit or will they all queue up and send at that rate?

I have never run into the SES sending limit. I have assumed that if more than 14 messages per second were sent by XF that SES would throttle those to 14 per second....although I'm guessing.

 

[GW2]

@mjda You can quit SES at any time. There is no contract. So, if SES does not meet your needs, just close your account. Amazon charges your credit card once per month based on your prior month's usage. And there is a section of the SES Dashboard that projects what your cost for the current month is going to be based on your usage to date for the current month.

 

[mjda]

Ok, I've been doing a lot of reading about this and still haven't been able to come up with a solution. I was hoping XF would add rate-limits on sending emails but after posts by @digitalpoint (here) and @Kirby (here) I don't think that's something that will, or even should, happen.

I tried looking into a way to do it at the MTA level on my server, but I wasn't able to figure out how to do that using Exim. Since I don't know enough about server administration to go tinkering with things that aren't provided via WHM or cPanel, I think it's best I go with another service, even though it was something I was hoping to avoid.

So, that brings me back to SES and I've got a few questions for some of you who may have used SES to send out bulk emails.

A few times a year I need to send out 1,000 emails using the bulk contact form in XF's admin panel. However, I end up getting blocked by Gmail, Yahoo and Hotmail for sending too many, too fast.

My first question is, will I not be sending the same number of emails in the same amount of time by switching to SES? It allows a 14 emails per second limit. What happens if I exceed that? Will SES still accept the emails and just mail them at that rate, or will all the emails, starting at 15, be dropped completely?

I already have email addresses set up for the bounce and unsubscribe so my next question is, will I still be able to make use of those options in XF? Also, will it work out of the box, once I have my SES email address set up? Is there some parameter that XF passes to SES to make it work?

Finally, right now I'm sending all XF emails from my user@mydomain.com email address. Will I still be able to use that email address for receiving mail and simply send mail using SES or will I need a completely separate email address for the SES setup?

 

[GW2]

@mjda This thread is has quite a bit of discussion about SES which might answer some of your questions:
https://xenforo.com/community/threads/amazon-ses-questions.208178/

Also this one:
https://xenforo.com/community/threa...ing-with-postmark-sendgrid-amazon-ses.213795/