Email best practices with spam filter concerns?

[OttoBomb]

Please move to correct area if this is posted in the wrong sub-forum.

I've been reviewing a few email options on my XF board and have had some serious problems with members receiving activation and notification emails, especially with Microsoft email services. I've contacted live.com, outlook.com, hotmail.com, and it has helped some, but enough are being blocked to be a problem.

Email delivery speed isn't a huge issue for me, successful delivery is. I'm wondering if SMTP and Google are a good option to keep me emails white-listed and ensure emails are consistently delivered. I'm willing to pay a few dollars a month for this, as long as I have a bulletproof delivery system.

Did anyone have a similar problem as I do, is so what did you do? I'm a bit new to configuring email systems, but can figure out anything as long as I have documentation for it.

In case you are wondering, yes I've read through the Email troubleshooting guide (https://xenforo.com/community/resources/email-troubleshooting-guide.5086), but I don't think it specifically address the problem I'm having.

 

[Sim]

This article by SparkPost may be useful - https://www.sparkpost.com/blog/6-rules-maximum-outlook-deliverability/

Microsoft addresses can be difficult.

I suggest using an ESP (Email Service Provider) - don't try sending directly via Google using SMTP, you'll run into issues and get throttled if you're sending too many emails.

I use SparkPost, but there's some threads on here from people using Pepipost and others.

 

[OttoBomb]

Thanks for the quick reply Sim. I'll take a look at the link now. Do you use SparkPost just for mass emails, or registration email also?

I have some 60,000 members and about 1.5 million posts, but I rarely send mass emails, the most I've ever sent is two in one year. I'm most concerned about my registration emails about which Microsoft is not happy. I do want to send out mass emails once a quarter, and I don't want any problems there to interdict my registration emails.

 

[Sim]

SparkPost are good in that they support both transactional emails (registration / password reset / etc) and newsletter emails and manage the reputation of these separately - however, their default is newsletter emails and you have to use the API (or modify the email header to inject SMTP headers) to tell the system you are sending transactional emails (I had to write a custom addon to do this for me).

You my find it easier with other providers.

SparkPost are pretty strict in insisting you adhere to their rules and policies - but they are one of the largest providers and have to work hard to maintain their deliverability reputation (which is one of the better ones out there from test data I've seen).

At the moment I'm using SparkPost mainly for just transactional emails.

You don't want to do something like send an email out to all 60,000 members at once, especially if you've never done a database cleanse.

I recently got a warning from SparkPost when I was sending emails to all 12,500 members on one of my sites where a large number hadn't been active in over 5 years and so bounce rates were really high (over 10%). Anything more than 2-3% average bounce rate and you're going to be on their radar - so you want to be careful.

There are lots of other providers out there though - so look around and compare Pepipost, Sendgrid, Mandrill (which I used to use before they changed pricing and became too expensive), Mailgun, Amazon SES, etc.

 

[eva2000]

with Sparkpost beware of what domain extensions are contained in your emails as Sparkpost can suspend your account for blacklisted domain names mentioned in your emails https://community.centminmod.com/threads/sparkpost-suspension.9122/

I posted a guide for my Centmin Mod LEMP stack users for better email deliverability at https://community.centminmod.com/th...ver-email-doesnt-end-up-in-spam-inboxes.6999/

Microsoft you want to have DMARC, DKIM, SPF all setup for your sending domain.

 

[Sim]

Yeah, like I said - really strict.

They introduced a new policy recently whereby they "manually" approve all sending domains (I suspect it's just a script, but still), and there must be a valid website with real content for that domain (plus a heap of other arbitrary rules).

I had set up my servers to send emails (output of cron tasks typically) via SparkPost using Postfix as a sending MTA, and they were set to send with addresses like root@web1.example.com ... but there's no website behind that domain (actually there is, but it's a simple "Welcome to web1" page), so they didn't approve it when I went to set up a new server using the same pattern.

I was pretty grumpy at first, but when I thought about it from their point of view, it does make sense when you're trying to stop people abusing your system and sending spam thus damaging the reputation of everyone else who is doing the right thing.

I solved the issue by reconfiguring Postfix to send using a real email address (basically I dropped the subdomain), so my server emails now come from an address that looks like: webmaster@example.com

I think SparkPost are really struggling to deal with the massive influx of new signups who are moving across from Mandrill (which was very lax in its rules and thus very easy to use and abuse) - there will be a lot of people taking advantage of their free sending to abuse the system or at least, send emails which are less than optimal and thus are getting marked as spam by increasingly overzealous ISPs and mail providers trying to keep inboxes spam free.

So while I find SparkPost's policies to be rather draconian - I do understand where they are coming from and I'd rather have a system which works really well if you can adhere to their strict rules, when the alternative is to have an increasing number of your members have endless difficulty receiving emails from you.

It's become increasingly apparent to me (based on my recent experiences) that maintenance of your sending lists and keeping control over your sending reputation is critical for ensuring deliverability.

 

[OttoBomb]

All very helpful, thanks.

I must say I'm much less concerned with mass mailers than I am with successful registration emails getting through. It appears Google with SMTP might be best suited for those registration emails, and also small batches of emails to users occasionally. I'm fine with sending out even a few hundred at a time, as long as my users can register with high success rate. Are there other issues concerns associated with Google+SMTP that I am not aware?

From what I can tell, ESPs are more focused on mass email efforts. I'm totally fine with sending out very small batches, or even not sending any out at all, so long as the registrations get through. Is there another, better solution than Google+SMTP for my use case?

 

[Sim]

Are there other issues concerns associated with Google+SMTP that I am not aware?

You'll need to turn on "insecure apps" for the account you intend to send through.

From what I can tell, ESPs are more focused on mass email efforts.

I'm not sure this is entirely accurate.

Yes, bulk email is an important part of what they are doing (because customers demand it), but transactional email (which wasn't even a thing 5 years ago), is increasingly important as demonstrated by SparkPost's quite sophisticated suppression list mechanism which is smart enough to know that just because a user has unsubscribed from a mailing list doesn't mean they shouldn't still receive transactional email.

Indeed many of the ESPs I've dealt with are much more focused on transaction email than bulk email.

 

[OttoBomb]

So is the Google+SMTP a viable solution for solid transactional email delivery I'm looking for? I'll keep in mind things like the "insecure apps", DMARC, DKIM, & SPF to ensure deliverability.

Forgive my continual, repetitive questions to Sim (and others), I'm trying to ensure the best configuration for my specific needs. The most recent thread I could find on Google+SMTP+XenForo was from six years ago, so I want to be sure I've got current info. I appreciate the help and knowledge.

 

[Sim]

So is the Google+SMTP a viable solution for solid transactional email delivery I'm looking for?

I have my HelpDesk and CRM system linked into my Google Apps accounts and have outbound emails delivered using Google's SMTP servers with no problems. But I'm only sending emails that we've written ourselves - no different to using a mail client, it's not automated (automated transactional emails from my helpdesk system go via SparkPost).

I would be reluctant to use the same for my forums because of the volume of automated emails (especially new thread/post notifications) could easily hit the limits of what Google finds acceptable.

You can only send 500 emails per day using Gmail (2000 per day for G Suite / Google Apps) ... so it's quite feasible you'll hit those limits with a moderately busy site.

The point is that Google's system is not designed to be used for high volume automated transactional emails - so I would recommend against trying to use it for that.

I guess if you've got a brand new site with very few members, you could start there while you research ESPs - but be prepared to move off the platform once you outgrow it.

 

[OttoBomb]

Cheers Sim. Really appreciate the info!

I'm certain we send more than 500 emails a day. I'll look into an ESP that fits my needs.

 

[Sachin Tiwari]

Cheers Sim. Really appreciate the info!

I'm certain we send more than 500 emails a day. I'll look into an ESP that fits my needs.

@OttoBomb We'd love to help you, check our offering https://xenforo.com/community/threads/100-million-free-transactional-emails-from-pepipost.120717/