Sending large number of batch emails

[Kirby]

I don't think that's something that will, or even should, happen.

Yup.

I tried looking into a way to do it at the MTA level on my server, but I wasn't able to figure out how to do that using Exim. Since I don't know enough about server administration to go tinkering with things that aren't provided via WHM or cPanel

I have no experience with any sort of server control panel software, but setting up Postfix to send emails via a smarthost isnt't thaat difficult.
But going with a mail service provider like Amazon SES is the right choice for most admins anyway.

My first question is, will I not be sending the same number of emails in the same amount of time by switching to SES?

You will be sending +- the same amout of emails / second to SES that you are sending right now.
I'd assume SES does proper queue handling to not overload recipient MX servers.

What happens if I exceed that? Will SES still accept the emails and just mail them at that rate, or will all the emails, starting at 15, be dropped completely?

Sorry, but I don't know that - maybe others who actually use SES know that?

I already have email addresses set up for the bounce and unsubscribe so my next question is, will I still be able to make use of those options in XF? Also, will it work out of the box, once I have my SES email address set up? Is there some parameter that XF passes to SES to make it work?

Should work just fine.

Will I still be able to use that email address for receiving mail and simply send mail using SES or will I need a completely separate email address for the SES setup?

Yes, should be possible.

 

[TPerry]

What happens if I exceed that? Will SES still accept the emails and just mail them at that rate, or will all the emails, starting at 15, be dropped completely?

You get throttled... which simply means SES sends back an error message and stops accepting email and it's up to your sending solution to catch that and start to resend from there (which XenForo can't do).
Honestly... for sending newsletters and such there are better solutions to choose to utilize than SES and the very basic XF ACP contact process.

 

[mjda]

You get throttled... which simply means SES sends back an error message and stops accepting email and it's up to your sending solution to catch that and start to resend from there (which XenForo can't do).
Honestly... for sending newsletters and such there are better solutions to choose to utilize than SES and the very basic XF ACP contact process.

This is what I was hoping was not going to happen. Looks like it's back to the drawing board for me...

The emails I'm sending aren't really newsletters. I need to contact a specific user group to let them know when things are ready for a new year. I just don't know of a better way to do it that will result in all the emails being delivered successfully and me not getting temporarily blocked every year.

 

[TPerry]

The emails I'm sending aren't really newsletters. I need to contact a specific user group to let them know when things are ready for a new year. I just don't know of a better way to do it that will result in all the emails being delivered successfully and me not getting temporarily blocked every year.

You can always set up your own Postfix MTA server, have your site send to it, then relay via Amazon SES.
You can throttle the outbound emails that Postfix will relay to a the amount that SES accepts, and Postfix itself will keep those you sent to it in queue until they all get sent. It's more manual work, but it should be a viable solution. The other option is on those rare instances you need to send them, sign up for a month of a paid service, send via it then cancel.

 

[mjda]

You can always set up your own Postfix MTA server, have your site send to it, then relay via Amazon SES.
You can throttle the outbound emails that Postfix will relay to a the amount that SES accepts, and Postfix itself will keep those you sent to it in queue until they all get sent. It's more manual work, but it should be a viable solution. The other option is on those rare instances you need to send them, sign up for a month of a paid service, send via it then cancel.

According to @digitalpoint if I used Postfix I'd be able to limit them myself and just keep using my own server to send them. As I said before, though, I'm not comfortable enough that I know what I'm doing to install and configure it. May just have to find another service that'll work and roll with that. Like you said, I could just use it for what I need and then cancel it afterwards. Not ideal, but better than having my server's emails blocked for several days.

 

[TPerry]

According to @digitalpoint if I used Postfix I'd be able to limit them myself and just keep using my own server to send them.

Depends on your deliverability... if you have good deliverability already, then why worry? The point of using SES as the relay is to improve that deliverability. If you already have an MTA solution that you can actually get into the guts of PostFix and modify it's config files, then you are already 1/2 way there.

 

[mjda]

Depends on your deliverability... if you have good deliverability already, then why worry? The point of using SES as the relay is to improve that deliverability. If you already have an MTA solution that you can actually get into the guts of PostFix and modify it's config files, then you are already 1/2 way there.

My deliverability is great...I haven't had a complaint of missing emails in years...until I try to send 1,000 emails at once. Then I'm temporarily blocked on Gmail and Yahoo for several days. I don't need a mail server...I just need a way to not try to send all those emails at once. Evidently that can be done with Postfix, but I don't have the knowledge to do it. So, my only option is to keep sending the emails as I have and get myself blocked for a bit or pay for a service that can do it for me. SES isn't gonna work if they're just gonna drop the emails anyways.

Also, my server doesn't have Postfix installed. My server uses Exim. I use WHM/cPanel, so if it can't be done in that...I probably can't do it.

 

[TPerry]

My deliverability is great...I haven't had a complaint of missing emails in years...until I try to send 1,000 emails at once. Then I'm temporarily blocked on Gmail and Yahoo for several days.

That could also be because you are sending from an IP that doesn't have enough history with them therefore they throttle it, which SES may not have an issue with since they are a "known" source.
The problem you may have is shoving a ton of emails at once to SES, then overloading the number of emails sent per second. XF is not really set up to be sending mass-emails with all the throttling/blocking that ISP's tend to do. It's more set up for transactional email delivery, which SES works fine for with almost every person using it. When you want to send out mass-emails reliably, there are services that offer that, and a reason that those services exist (exactly what you are dealing with).

 

[mjda]

Ok, so I found out that when sending large chunks, you're allowed to go over the rate limit in bursts, but not for sustained periouds. So, I'm thinking I should be fine with what I'm wanting to do. So, I decided to go on with Amazon SES to give it a shot. Worst case scenario I'm just right back to where I started in a few months.

Trying to get it working today, though, I keep getting the following error when sending an outbound test email:

Connection could not be established with host email-smtp.us-east-2.amazonaws.com :stream_socket_client(): Unable to connect to tcp://email-smtp.us-east-2.amazonaws.com:587 (Connection refused) (code: 0)

I am still in the sandbox, but I have verified both the email I'm sending from, the domain I'm sending from, and the email I'm sending to. I've removed, and re-entered, the SMTP details numerous times so I'm sure they're correct. I just can't figure out what's going on here so any help I can get would be much appreciated!

 

[Anthony Parsons]

From memory, I had to contact them to remove me from the sandbox and told them estimated daily sending. As long as you keep SES bounce and complaint rates under their threshold, you are fine. If you send bulk email that many are marking as spam or bouncing, ie. unverified list / spam list, SES will ban you quickly.

 

[mjda]

From memory, I had to contact them to remove me from the sandbox and told them estimated daily sending. As long as you keep SES bounce and complaint rates under their threshold, you are fine. If you send bulk email that many are marking as spam or bouncing, ie. unverified list / spam list, SES will ban you quickly.

Will removing me from the sandbox fix my issue? From what I'm reading, I should have access to all features, but with a limited send rate. I've only sent 1 email so far, using their send test email option. It worked fine. I'm wondering how to send emails from SMTP.

 

[mjda]

Actually, I don't think that's the issue at all. I just connected my PC to send an email and it went through just fine. So, I'm thinking Amazon has my server blocked for some reason. Either that or I've got some kind of firewall issue preventing me from connecting to their server. About to contact Knownhost now to see if they can figure out what's going on.

 

[mjda]

Yep, turns out my firewall was blocking all outgoing SMTP connections. Kind of a weird error to get back (connection refused), but Knownhost was able to figure out the issue and show me how to fix it so I don't have that issue again.

Just requested to get out of the sandbox. Assuming that happens, we'll be sending on SES soon.

 

[mjda]

Well, that didn't work out. Amazon SES denied me saying I may lower their reputation. I'm not sure how they come up with that, but oh well. On to another solution.

 

[Anthony Parsons]

I used a third party to clean my database user preferences before I switched to SES. Once I had a recent database of accounts with updated preferences to block email based on complaints and bounces, I switched to SES, gave them all the details that I had cleaned up old account preferences and such, told them exactly how I would then use the email going forward, and to this day I ensure all bounces and complaints are handled correctly to send no more.

I have new users who get email, send it to spam, then complain because they can't get email updating them about new posts to their content. I tell them why, they make some lame excuse, I give them the benefit of doubt, unblock it, but the moment they do it twice, their laziness between delete or junking it, it remains blocked after that.

I would rather not upset Amazon and have my email blocked. If you want to use them, clean it up via another company, then tell Amazon and hope it works.

 

[spk100]

I made the mistake of sending out emails from my server (3 days ago) and I got listed in the spam database which got me in the bad books of my host.