Amazon SES and Xenforo

[Pieman]

Hi all,

Looking to move my Xenforo transnational email from the local host to SES. Reading on here, I need to apply for a message limit to remove me from sandbox mode, this will then give me a peak number of messages per day and per month, but it also has a peak rate limitation as well, and if you deliver messages faster than that amazon just drops them.

So my question is what is the best way to have SES set up with Xenforo, how can I ensure that I don't exceed the max delivery rate for messages, on highly followed threads for example? Is there a guide to how to set up SES or am I best getting an addon?

I also send out an eshot to members of around 30k, the list is cleaned for unsubscribes, but bounces have not been cleaned (roughly 1.2k), what would be the best way to get his list SES fit?

thanks in advance.

 

[fly]

First things first, you've got to clean up that bounce list. Amazon will not tolerate that many bounces. After that, you should be fine. Are you using SES directly or something like Sendy?

 

[Pieman]

Thanks @fly

The bounce list is only in relation to my eshot emails that I send via a different email server for which bounces don't matter (not with Sendy but I will look at that) and can continue to do however I do want to clean it, I have long lists of bounce emails in a mixed email account, but no obvious automatic way to clean the list and I'd rather not go in and edit each user that has a bounce email. The bounces from the forum transaction emails are only a handful a week mainly due to fat fingers on sign up, so I thought that would be a good way to start using SES with a view to getting proper stats on openings, spam etc.

I don't have Sendy at the moment only an SES account, and even though I have read most of what I can find on the forums I can't find an 'idiots' guide to how to implement SES without the possibility of loss of some emails due to throttling limits.

 

[fly]

IIRC, it's pretty straightforward, get out of the sandbox, validate both your sending address and bounce address, get SMTP credentials, and plug them into xF. I've had a few clients use SES and don't recall any issues with rate limiting, but you could always install something like this:

https://xenforo.com/community/resources/email-queuing-enhancements.3944/

 

[Pieman]

Thanks, I'll get out the sandbox and then change over when the forum isn't busy. I'm probably making too much of it.

EDIT: requested a new limit so hopefully I can change over at some time this weekend.

 

[Anthony Parsons]

this will then give me a peak number of messages per day and per month, but it also has a peak rate limitation as well, and if you deliver messages faster than that amazon just drops them

Amazon do not state what you are. Where are you getting such information?

http://docs.aws.amazon.com/ses/latest/DeveloperGuide/limits.html

Amazon sandbox is your initial environment. You put in a request for how many emails you need to send per 24hrs, Amazon will then give you the required daily limit to meet your needs. My account has a 75k per day limit on it.

The only hard limit is 10Mb per message. Amazon do not discard or drop email you send.

I've always used this service to clean my lists: https://serversmtp.com/en/cart.php?systpl=turbo-smtp with the pay as you go service, a one time send to clean bounces.

 

[Mouth]

this will then give me a peak number of messages per day and per month, but it also has a peak rate limitation as well, and if you deliver messages faster than that amazon just drops them.

Amazon doesn't just drop them, it return a throttling error to your email client. I'm not sure how XF handles SMTP send errors, it would be worth asking.
Refer: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/reach-sending-limits.html

My send rate is 14 emails/second. I've never had any problems with this rate.

 

[Pieman]

Thanks @Mouth and @Anthony Parsons , it did seem odd that they dropped them but that was what I had read on various threads... Amazon upped my limit to 50k a day (I only asked for 30k) which gives the same 14 emails/second that you have had no trouble with. I have set DKIM up so it's all looking good to go.

I will get my newsletter list cleaned up as with this new limit I can send that using SES and really get into how many of my emails are being read/junked.

thanks all for your help.

 

[Anthony Parsons]

The best thing with SES is that you get the default 2+ email score that Amazon give, and adding Amazon DKIM records as CNAME records to your domain, just boosts that a little more. Combine their SNS topic bounce and complaint email handling that you can add into your sending email, diverting anyone who marks your XF mail as spam or bounce back to your XF bounce handler, subsequently placing the members account into an unverified state.

 

[Pieman]

It seems to have gone smoothly apart from my copy and pasting a username rather than the amazon smtp server into the email settings... Email is going out nicely thanks everyone.

 

[Mouth]

Combine their SNS topic bounce and complaint email handling that you can add into your sending email, diverting anyone who marks your XF mail as spam or bounce back to your XF bounce handler, subsequently placing the members account into an unverified state.

I'm always having problems with this, despite adjusting SES/SNS several times, but XF Email Bounce Log showing the type as Unknown and action None.
Could you screenshot and share your SES config for this please?

 

[Pieman]

I'd be interested in that as well, I have set up a SNS for bounce etc, as for some unknown reason my normal bounce emails were disappearing even though test ones came fine. It seems that you could just collect all the bounce emails in a google mail box and setup xenforo to use that. So is the bounce system not parsing the Amazon bounce consistently or at all @Mouth?

 

[viper357]

I've always used this service to clean my lists: https://serversmtp.com/en/cart.php?systpl=turbo-smtp with the pay as you go service, a one time send to clean bounces.

Hi, how exactly would you clean your lists with this? Just by sending all your emails through this service? But does the built-in automated bounce processing in Xenforo not do the same thing?

 

[Pieman]

Out of interest just read this

Maximum Send Rate—The maximum number of emails that Amazon SES can accept from your account per second. You can exceed this limit for short bursts, but not for a sustained period of time.

in the sending limits documentation. They don't define brief of course but even if it was only a few seconds that could allow a few hundred emails to be processed, and that should be easily enough for me even on my longest threads.

 

[Mouth]

So is the bounce system not parsing the Amazon bounce consistently or at all @Mouth?

For me, inconsistently. It will seems to work for a little while, then stop working. It doesn't start working again until I make some change to SES/SNS.

 

[Mouth]

even if it was only a few seconds that could allow a few hundred emails to be processed

Is XF even processing emails that fast to SMTP? I understand email is sent via the deferred queue, and the email sending action has a qty limit per deferred cycle.
Would need to confirm that viewpoint though. Not had issues with email sending rate, so never looked deeply.

 

[eva2000]

doesn't matter Amazon SES automatically monitors when you start going over the preset rate limit and automatically raises you up to get rate limit level if you consistently are reaching the previous preset anayway

 

[Pieman]

For me, inconsistently. It will seems to work for a little while, then stop working. It doesn't start working again until I make some change to SES/SNS.

That is annoying, currently I am getting all delivered email events sent to me via SNS, looking at the headers I am not getting anywhere near the 14/sec limit so that is good. Next step is to get Sendy set up and clean up my newsletter list...

 

[Mouth]

For me, inconsistently. It will seems to work for a little while, then stop working. It doesn't start working again until I make some change to SES/SNS.

That is annoying

FYI. Was spurred to investigate again this evening. My bounces mailbox is with gmail. It was sporadically sending my SES/SNS bounces notifications to the spam folder. Arrgh. Have setup a filter on gmail now to never send them to spam.

 

[Pieman]

thanks for that, my bounce mailbox will be a gmail one as well so I will set that up from the start. Not that I am getting any bounces as I said my transactional mails are a very clean list. Out of interest what settings do you use on for bounces with xenforo I see the default is 3 which seems too high to me, I was thinking of using 1.