XF 1.1 Amazon SES With Xenforo

[TheBigK]

Yesterday, I stumbled upon Amazon SES which now has SMTP gateway. I'm planning to use it with Xenforo to handle all the emails; so that the delivery is always right. I, however have basic questions -

• To the best of my knowledge, I only need to enter the SMTP settings provided by Amazon in 'Email Options' in XF ACP. Is that enough?
• Can I use it to send notifications or monthly newsletter to all the members who have opted to receive emails from administrator? Would that be spamming? We've over 90k members who've opted to receive emails from admins; but my estimate is that 10-20k of them aren't active. We didn't have 'email' verification enabled in our earlier days.

Thank you for your time in advance

 

[PJK]

Thanks a lot for the info and your time.

Right now I'm still struggling to simply get the connection to the bounced email (I'm getting this error below or the cannot connect error depending on which settings I use):
cannot read - connection closed?10 minutes ago - library/Zend/Mail/Protocol/Imap.php:139
I've tried IMAP and POP3 with 995 and 993 ports, and TLS and SSL, all still giving me errors. The username and password is definitely correct as I reset the password an hour ago to ensure that is the case. I've tried all the combinations I can think of. The only other variable is the Host name, I'm using host.domain.com. I've also tried with the IP of the server and still no luck. I am using Cloudflare on domain.com. Any other ideas?

 

[Anthony Parsons]

I'm using host.domain.com

Why aren't you using mail.domain.com? That is typically the correct format for mail.

 

[PJK]

Why aren't you using mail.domain.com? That is typically the correct format for mail.

I've tried domain.com, host.domain.com, and mail.domain.com. Getting errors still. Depending on the settings of TSL/SSL, IMAP/POP3 and port, I get one of these 2 errors:

1. cannot connect to host; error = (errno = 0 )23 minutes ago - library/Zend/Mail/Protocol/Pop3.php:109
2. read failed - connection closed?53 minutes ago - library/Zend/Mail/Protocol/Pop3.php:174

I've been talking to the host and they said the settings are correct. Is the fact that the use Cloudflare for domain.com the issue? I'm unsure where to go from there to fix the connection issues.

 

[Anthony Parsons]

I use cloudflare... no issue, as mail bypasses their system.

 

[Mouth]

Complaints coming from Amazon SES are not being handled by XF ...
I wonder if people think they should be? Eg. Changing user state to Awaiting Confirmation (from edit)

Delivered-To: bounces+36aa4aae+coolj<redacted>
Received: by 10.202.229.14 with SMTP id c14csp1687307oih; Mon, 11 Apr 2016
16:02:11 -0700 (PDT)
X-Received: by 10.66.63.104 with SMTP id f8mr36354299pas.109.1460415731058;
Mon, 11 Apr 2016 16:02:11 -0700 (PDT)
Return-Path: <no-reply@amazonses.com>
Received: from a27-9.smtp-out.us-west-2.amazonses.com
(a27-9.smtp-out.us-west-2.amazonses.com. [54.240.27.9]) by mx.google.com with
ESMTPS id 14si5921053pfk.156.2016.04.11.16.02.10 for
<bounces+36aa4aae+coolj<redacted>> (version=TLS1
cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 11 Apr 2016 16:02:11 -0700
(PDT)
Received-SPF: pass (google.com: domain of no-reply@amazonses.com designates
54.240.27.9 as permitted sender) client-ip=54.240.27.9;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of
no-reply@amazonses.com designates 54.240.27.9 as permitted sender)
smtp.mailfrom=no-reply@amazonses.com
X-Original-To: complaints@email-abuse.amazonaws.com
Delivered-To: complaints@email-abuse.amazonaws.com
Date: Mon, 11 Apr 2016 23:02:10 +0000
From: complaints@us-west-2.email-abuse.amazonses.com
Subject: complaint about message from 10.169.62.154
To: bounces+36aa4aae+coolj<redacted>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="88B58A19-DB59-4477-93F6-F3D0EDB29B11"
Message-ID: <01010154078f91a1-6ba9ba12-0039-11e6-8407-e7a9e74ce211-000000@us-west-2.amazonses.com>
X-OriginalArrivalTime: 11 Apr 2016 23:02:08.0665 (UTC)
FILETIME=[2C8F1090:01D19446]
X-SES-Outgoing: 2016.04.11-54.240.27.9

 

[Anthony Parsons]

Yer, I don't send complaints to XF, but to my email account, as they're often lazy users who don't want to login and change their settings, so when I get a complaint response I can action it myself.

Sendy I do, as it handles complaints and bounces directly, but I use a different email for that than XF.

 

[PJK]

Create an account with a nonsense email and send a single message to it so that it bounces. The bounce should be in the account. Ensure you have the bounce email account open, as once XF processes the account it deletes the email.

If the bounce email appears, go and manually run the cron to process and you will see the account automatically change status to an unconfirmed status due to email.

Bounces and such can be setup through SNS, especially if you're sending directly from xenforo you can send them direct to your bounce account for automatic processing. Saying that, there are soft and hard bounces, so this can cause issues for regular members of your community.

If you're sending via an email service or software, it should have instructions for bounce handling so that they're handled better than what XF does by default, which is a simple system of bounced, or not.

I use Sendy on my server to send everything via SES, with all handling done automatically back to Sendy via SNS. It simply syncs with XF accounts appropriately.

So the errors connecting the account I couldn't fix with my server, so I just setup an email bounceddomain@gmail.com, and using Gmails IMAP settings I no longer got errors when running the cron. Bingo, next step.

After Enabling the Automatic Bounce handler, I created an account on the forum with a nonsense email, then sent a message to that user. Checking the bounceddomain@gmail.com continuously to see if any emails showed up. Nothing showed in the inbox, but the "Delivery Status Notification (Failure)" did end up showing in the Spam box. I then marked the message as Not Spam, which put it into the inbox.

Then I ran the cron. I watched the email get deleted from the inbox. When I checked the user, their User State did not change, it still remains as Awaiting Email Confirmation.
a) Why didn't the user state change when the email was deleted, and how do I solve this?
b) How do I solve the issue with the bounced emails going to Spam? Is it something I need to fix or will the handler look in the Spam box for bounces?

Thanks.

 

[Mouth]

Then I ran the cron. I watched the email get deleted from the inbox. When I checked the user, their User State did not change, it still remains as Awaiting Email Confirmation.
a) Why didn't the user state change when the email was deleted, and how do I solve this?

/admin.php?logs/email-bounces

b) How do I solve the issue with the bounced emails going to Spam? Is it something I need to fix or will the handler look in the Spam box for bounces?

Go into your Gmail settings for the mailbox, and whitelist *.amazonses.com

 

[PJK]

/admin.php?logs/email-bounces

It shows it as a Hard Bounce:
https://www.dropbox.com/s/yacyecy0lzopez1/Screenshot 2016-05-03 17.21.01.png?dl=0
Shouldn't the user state change after this hard bounce?

 

[Mouth]

It shows it as a Hard Bounce:
https://www.dropbox.com/s/yacyecy0lzopez1/Screenshot 2016-05-03 17.21.01.png?dl=0
Shouldn't the user state change after this hard bounce?

I would expect so. I'd suggest a support ticket with XF.

 

[Paul B]

Users that don't have the valid state won't receive conversation or thread emails so the bounce status is irrelevant in this context.

 

[Mouth]

Users that don't have the valid state won't receive conversation or thread emails so the bounce status is irrelevant in this context.

You're suggesting that in @PJK's case the gobbledeedook@gmail.com user state likely wasn't 'valid' (yet?) and thus the hard bounce recorded in the bounced log wouldn't have resulted in an action on the user state?

 

[Paul B]

You're suggesting that in @PJK's case the gobbledeedook@gmail.com user state likely wasn't 'valid' (yet?)

That's what he stated above:

Then I ran the cron. I watched the email get deleted from the inbox. When I checked the user, their User State did not change, it still remains as Awaiting Email Confirmation.

Unless I misunderstood.

 

[Anthony Parsons]

a) Why didn't the user state change when the email was deleted, and how do I solve this?
As above, you left the account in awaiting confirmation status, which means the account isn't verified anyway. The account needs to be set to verified for it to change.

b) How do I solve the issue with the bounced emails going to Spam? Is it something I need to fix or will the handler look in the Spam box for bounces?
As above, whitelist the amazon address in gmail with a wildcard entry.

 

[PJK]

a) Why didn't the user state change when the email was deleted, and how do I solve this?
As above, you left the account in awaiting confirmation status, which means the account isn't verified anyway. The account needs to be set to verified for it to change.

b) How do I solve the issue with the bounced emails going to Spam? Is it something I need to fix or will the handler look in the Spam box for bounces?
As above, whitelist the amazon address in gmail with a wildcard entry.

Users that don't have the valid state won't receive conversation or thread emails so the bounce status is irrelevant in this context.

Ah, so only users with a Valid user state will work, I didn't know that. I manually changed the test account with the nonsense email to Valid, then emailed them, and it changed the state of the account correctly, thanks!

For users who register though with an invalid email, they remain in awaiting activation. How do I deal with these users?

 

[Paul B]

Nothing much can be done if they use an invalid email address to register with.

I typically check and delete them at the end of the month.

 

[Anthony Parsons]

I have an old addon that automatically cleans them up every 14 days. It sends them an email to confirm their account during the process, and if it doesn't get activated, automatically deleted.

https://xenforo.com/community/resources/8wayrun-com-xenutiles-pruner.2152/

Still works a charm in the current XF version.

 

[PJK]

I have an old addon that automatically cleans them up every 14 days. It sends them an email to confirm their account during the process, and if it doesn't get activated, automatically deleted.

https://xenforo.com/community/resources/8wayrun-com-xenutiles-pruner.2152/

Still works a charm in the current XF version.

Thanks for that recommendation, I'll use it.

So I did an email shoot today to the 30,000 members of the forum (I haven't done a mailing in nearly a year, and in the past did it using vB so unsure of bounce rates and delivery rates). A couple things:
1) It took several hours to send out. Is this normal? It kept processing through all the users. Is there a way to speed it up?
2) I got an email from Amazon SES about being in probation:

The Amazon SES sending for AWS account xxxxxx, which lists you as a contact or owner, has been put on probation in AWS region US East (N. Virginia) because it has triggered alarms for excessive bounces. You may still send mail with the account, though we ask that you take action to address this problem as soon as possible.

Your current bounce rate is 11.25%. This is measured over the last 10,376 eligible emails you sent, spanning over approximately the last 11.1 days. We expect our senders' bounce rates to remain below 5%. Senders with a bounce rate exceeding 10% risk suspension.

Note that the bounce rate includes only hard bounces, and excludes bounces to domains you have verified.

Your account will be on probation for the next 42,000 emails you send.

At the end of the probation, if your bounce rate has not returned to acceptable levels, your ability to continue sending via Amazon SES will be suspended. Please note that if your bounce rate increases further, your sending abilities may be terminated before the end of the probation period.

If your bounce rate improves to acceptable levels and maintains that status, you will be notified when the probation is lifted.

Look at the Email bounce log, it shows 87 pages of hard email bounces. I'm guessing this is from the 10 years the site has existed and the first time I've seen data from bounced which should clear them up. I assume the automatic email bouncer will fix this issue so all those people will be removed. I'm unsure why so many bounced considering you have to verify your email to post on the forum.

How do you recommend going forward from here for mailings to prevent suspension of SES?

Edit: Also, Xenforo said I sent out to 30,000 members, but on Amazon SES it shows 10360 emails sent. What's the discrepancy caused by?

Edit2: Just noticed there are 1554 pages of server error logs:
Email to domain@user.com failed (after retry): Message rejected: Sending suspended for this account. For more information, please check the inbox of the email address associated with your AWS account.Today at 10:14 PM - library/Zend/Mail/Protocol/Abstract.php:431
-----
So, looks like less than half of the emails got sent to the high bounce rate. How do you recommend handling this so the rest of the emails go out okay?

 

[Anthony Parsons]

How do you recommend going forward from here for mailings to prevent suspension of SES?

Amazon requires you to use clean lists... they don't tolerate any nonsense, basically. You need to ensure that whatever you're doing, that 11% of users that bounced are not sent email again.

I don't use XF to send email, I use Sendy, as it controls this. I also had a clean list before ever going to SES. Use another SMTP provider to clean your list if needed, a single send and then copy over the legit list that didn't bounce as your current subscriber list.

You can't screw with Amazon... you need to ensure your email list is active and clean, that you're not sending email to users from years ago or such. Amazon tell you all these things in their info and terms.

 

[PJK]

Amazon requires you to use clean lists... they don't tolerate any nonsense, basically. You need to ensure that whatever you're doing, that 11% of users that bounced are not sent email again.

I don't use XF to send email, I use Sendy, as it controls this. I also had a clean list before ever going to SES. Use another SMTP provider to clean your list if needed, a single send and then copy over the legit list that didn't bounce as your current subscriber list.

You can't screw with Amazon... you need to ensure your email list is active and clean, that you're not sending email to users from years ago or such. Amazon tell you all these things in their info and terms.

Since it only sent the first 10k (which are probably the older members), the remaining 20,000 are probably a much lower bounce rate as they are newer members. Is it possible to have the email sent out to the *only* remaining people using just the default email transport method (my server)? Or do I need to re-send out to everyone again?

I assume the first 10,000 are cleaned now that hard bounces were removed. Can you recommend other SMTP providers to clean the list? Thanks.