Amazon SES and max emails per second

[marcv]

Is anyone using Amazon SES for email? My current send rate is 5 emails per second (max 10k). The max per 24 hours is 10k which is fine for now.

The 5 email per second has me concerned. Two scenarios:

1) A thread that is heavily subscribed to gets a reply. XF sends out notices to 50 subscribers of new post. If sending all at once will breach the 5 emails per second.

2) Newsletter mailing to all members. No delay will likely blow past 5 emails per second.

Questions:

1) Does anyone have a higher send rate than 5 per second?

2) How many emails is XF likely to attempt to send per second all out speed?

3) Is there any simple way to "throttle" the send?

4) I think I read that Amazon SES dumps any emails exceeding the send rate instead of queuing them. Stupid IMHO but if this is the case no choice but to increase send rate per second or devise a send throttle mod

thanks

 

[WSWD]

I obviously haven't looked at the XF code, but any decent script is going to throttle emails. Somebody from XF is going to have to answer your questions specifically, but I would be shocked if XF just blasts out all the emails at once.

 

[trizz]

Sending Quota: send 100000 emails per 24 hour period
Quota Used: 2% as of 2013-09-21 05:58 UTC-4
Max Send Rate: 28 emails/second

if you ask for a higher quota, they will give it to you.

 

[Floren]

Is anyone using Amazon SES for email? My current send rate is 5 emails per second (max 10k). The max per 24 hours is 10k which is fine for now.

Did you checked how many emails you send every day? logwatch will do that for you automatically, if you want a detailed report you can look at qshape. IMO, Amazon SES is used for sending large amounts of emails, (i.e. 1 million emails/day). Postfix can handle a HUGE amount of emails with its fairness and no thundering herd features.

 

[marcv]

Sending Quota: send 100000 emails per 24 hour period
Quota Used: 2% as of 2013-09-21 05:58 UTC-4
Max Send Rate: 28 emails/second

if you ask for a higher quota, they will give it to you.

Ok I asked and got up to 14 messages per second. However xenforo tries to send 30 or so per batch per second so half get dumped.

Can any of the xenforo coders give me the script where to insert a sleep function that delays the next batch (I think it is 30 per batch default on xenforo). Might be something where a counter is used and ever 10 emails the script waits 1.5 seconds until the next 10 are sent using modulus function to see if multiple of 10 if I recall

 

[trizz]

Ok I asked and got up to 14 messages per second. However xenforo tries to send 30 or so per batch per second so half get dumped.

Can any of the xenforo coders give me the script where to insert a sleep function that delays the next batch (I think it is 30 per batch default on xenforo). Might be something where a counter is used and ever 10 emails the script waits 1.5 seconds until the next 10 are sent using modulus function to see if multiple of 10 if I recall

im not pushing xF email thru SES yet, and likely wont. i migrated to mandrill a few weeks ago for deeper analytics for delivery, click thru, etc and couldnt be happier. now i can find out which specific user(s) have click the spam button and ask whats up.

i had no idea xF batches email in blocks of 30. youd think that would be a configureable option....

ill likely be pushing xF email thru mandrill in a few weeks

 

[Jim Boy]

Amazon SES is not well suited to large Xenforo installations. We normally clock up about 10000 emails a day without blinking and we were booted from there withing a couple of days because our bounce rate was too high. I'm glad we hadn't sent out one of our regular newsletters which can number around 100,000 in a day.

While it is cheap it is pretty much their worst service and I think they only do it cos 'they have to'

A better solution is to send email out from your xenforo installation, or another box within your VPC - which is what we do. Very important thought to get sending restrictions lifted. AWS will choke outgoing SMTP traffic by default as a protection against spammers. To get it lifted, go to https://portal.aws.amazon.com/gp/aws/html-forms-controller/contactus/ec2-email-limit-rdns-request and fill in the details.

 

[marcv]

My limit is 50k daily but that only comes with 14 emails/sec. Doing a newsletter seems to send well more than this number per second. At least 2x if not more. Amazon does not queue anything over the send rate.

I'm going to have to setup and maintain postfix or sendmail smtp on my server. Was hoping to avoid this and use SES.

 

[Mouth]

exim4-daemon-light is the best for a send only mail server, IMHO

 

[ehd]

For newsletters I would use this with SES: http://sendy.co/

Why don't you use your server for forum-emails?

 

[marcv]

For newsletters I would use this with SES: http://sendy.co/

Why don't you use your server for forum-emails?

Just easier to let Amazon handle it and not have to use our own server as long as AWS gives us both the amount of emails and the proper sending rate to keep up.

 

[ehd]

Actually I don't see a reason why I should pay for regular forum-emails (Amazon), when I can get that for free (own server).

Dealing with newsletters is another thing.

 

[Cyberclaw]

Is there a solution to this issue? Is there a way to slow down how many emails Xenforo sends out per second?

 

[marcv]

Is there a solution to this issue? Is there a way to slow down how many emails Xenforo sends out per second?

None I could find unfortunately. If you find something let me know though

 

[TPerry]

Actually I don't see a reason why I should pay for regular forum-emails (Amazon), when I can get that for free (own server).

Dealing with newsletters is another thing.

One reason... odds of delivery is a lot higher if your IP does not have good reputation. Maintaining a mail server so it doesn't become black listed and has a higher delivery rate is not without work itself (and sometimes cost).

 

[Cyberclaw]

None I could find unfortunately. If you find something let me know though

Well, I'm happy to report that the issue is resolved. I put in a request with Amazon SES to have the send rate limit increased. In the request I explained that we are using Xenforo, and that we need the rate limit increased to match Xenforo's send rate (appoximately 30 emails per second). Otherwise, I explained, our members will miss notifications that they expect to receive. Within a few hours Amazon SES responded, and increased our send rate limit accordingly.

 

[marcv]

Well, I'm happy to report that the issue is resolved. I put in a request with Amazon SES to have the send rate limit increased. In the request I explained that we are using Xenforo, and that we need the rate limit increased to match Xenforo's send rate (appoximately 30 emails per second). Otherwise, I explained, our members will miss notifications that they expect to receive. Within a few hours Amazon SES responded, and increased our send rate limit accordingly.

Wow! That’s awesome. Way back when I had requested same from them they refused and said the rate would not increase until our experience rating had more time. I explained I could never get more time because XF would error out being unable to send at the required rate. Chicken and the egg problem. Glad to see these days they are more flexible! Thx for update!

 

[Anthony Parsons]

We normally clock up about 10000 emails a day without blinking and we were booted from there withing a couple of days because our bounce rate was too high.

There should be no reason Amazon would boot you IF you're actively processing bounces at your install... and considering XF has this built-in, there is just no reason not to be, thus your user sending list should be super clean all the time, Amazon or not.

 

[snoopy5]

Did anything change since the last posting here?

I use X 1.5.23 with Amazon SES. In my Amazon overview, I have a sending rate of 14 emails per second. But in reality, it takes a lot longer until all emails are sent out.

I wonder whether this is due to some changes within XF so that XF gives less emails out to SES than SES could handle per second.

A mailing with >22K users took over 2 hours. But if you calculate it with 14 emails per second, it should not have last longer than around 27 minutes.

Bounce rate is only 0,12%. Mostly mailbox full alert as the reported reason for the bounces. Only one complaint. So this can not be the reason.

I try to find out now why it took so long. This is a dramatic difference. 2 hours vs. 27 minutes.

Does anybody has an idea what the reason could be for that? Is there something in the code of XF, that I can change so that it sends more emails per second to Amazon SES to match those 14 emails per second?

 

[snoopy5]

anybody?