Intergration with off-site e-mail services (Mandrill, Amazon SES, etc...)

[nrep]

For larger forums, sending e-mail via Mandrill, Amazon SES and other e-mail services can be very useful. It offloads work from the server and provides a more reliable method of mail delivery (as it is sent from managed, trusted IPs).

These services also provide excellent bounce handling, which can be integrated via simple webhooks.

There is an excellent addon already that does this, but it is so useful that it could really be a core feature.

 

[thedude]

We send via Mandrill. We built our own add-on that uses webhooks for automated bounce handling (like you mentioned), and also to be notified when a member reports our mail as spam.

Our addon also lets us specify Mandrill subaccounts, so all of our sites' emails are cleanly categorized by site whenever we need to log into Mandrill.

We may release it to the community here, though I think a couple of existing addons do something very similar to ours.

+1 if some of these features could be generalized and added to the XF core.

 

[nrep]

Nice work @thedude - I would be extremely interested if you release this addon!

 

[Mert]

We send via Mandrill. We built our own add-on that uses webhooks for automated bounce handling (like you mentioned), and also to be notified when a member reports our mail as spam.

Our addon also lets us specify Mandrill subaccounts, so all of our sites' emails are cleanly categorized by site whenever we need to log into Mandrill.

We may release it to the community here, though I think a couple of existing addons do something very similar to ours.

+1 if some of these features could be generalized and added to the XF core.

I was checking the other mandrill integration however yours sound way more compatible with our needs. If you ever plan to release , feel free to notify me

I also would like to see this covered with Xenforo 2.0

 

[nrep]

Has anyone been able to get Amazon SES integration working with XenForo 2.0? I was really hoping this would have made it over, as it handles bounces so well.

 

[Alpha1]

I'm amazed that this suggestion has not received more traction. Sending out email has become an issue that many webmasters struggle with.
Just like the Generic Payment Framework in XF2, I suggest a Generic Email Framework to provide integration with services like:

• Elastic Email
• Sendgrid
• SendinBlue
• Mailgun
• Amazon SES
• Sparkpost
• Mandrill
• Mailjet

This would optimally include integration in regards to:

1. bounced email
2. list cleaning
3. mail templates
4. analytics

 

[S Thomas]

Xenforo uses Swiftmailer as the underlying email handler. As such you would typically look for the respective Swiftmailer integration for your email service. After that you could use the composer tutorial from @Sim to integrate the new lib into XF.
Examples:
https://packagist.org/packages/expertcoder/swiftmailer-send-grid-bundle
https://www.mailgun.com/blog/php-how-to-use-mailgun-efficiently
https://packagist.org/packages/slowprog/sparkpost-swiftmailer

With that being said, I guess most of the stuff you want to get handled is already built in. Mail templates and bounced email handling are part of the core if I'm not mistaken. And part of bounced handling is to cleanup lists if that's what you are looking for. However, I don't know if or how that is handled by external libs.

 

[Mouth]

integration with services like

Google mail too, using https://developers.google.com/api-client-library/php/auth/web-app and https://developers.google.com/gmail/api/quickstart/php

 

[Sim]

Google mail too, using https://developers.google.com/api-client-library/php/auth/web-app and https://developers.google.com/gmail/api/quickstart/php

The problem with Google mail is the limitations they place on sending - Gmail isn't intended for high volume mail sending.

The Gmail API enforces the daily mail sending limits described here: https://support.google.com/a/answer/166852

Basically, 2,000 messages per day is a hard limit which cannot be changed or worked around. Should be find for smaller forums - but completely unsuitable for larger sites.

 

[Sim]

Xenforo uses Swiftmailer as the underlying email handler. As such you would typically look for the respective Swiftmailer integration for your email service. After that you could use the composer tutorial from @Sim to integrate the new lib into XF.
Examples:
https://packagist.org/packages/expertcoder/swiftmailer-send-grid-bundle
https://www.mailgun.com/blog/php-how-to-use-mailgun-efficiently
https://packagist.org/packages/slowprog/sparkpost-swiftmailer

With that being said, I guess most of the stuff you want to get handled is already built in. Mail templates and bounced email handling are part of the core if I'm not mistaken. And part of bounced handling is to cleanup lists if that's what you are looking for. However, I don't know if or how that is handled by external libs.

Just be aware that SparkPost does not return bounced messages to the sender, so the built in bounce handling doesn't work with SparkPost. They send bounces as webhook notifications, or you can poll their servers for bounce notifications - requires development.

I actually have built a fully functioning SparkPost addon (uses this library: https://github.com/f500/swiftmailer-sparkpost) with full bounced email support - but it's not quite ready for release yet and will be a paid addon.

 

[Alpha1]

However, I don't know if or how that is handled by external libs.

How it is handled by external providers is a key factor. i.e. issues with conflicting mail templates, feedback loops, handling bounces beyond status codes. For example with some provider you get a double footer because the XF template gets wrapped in the email provider template. Which makes no sense and needs sorting out.

Just like XF did for Stripe in the payment framework, I'd like to see one external email provider implemented in core so that such issues are encountered.

@Sim how would you like to see this suggestion implemented? How could XF make integrations with external email providers much easier?

 

[Sim]

I'm not sure more work really is required beyond what is already provided in XF2.0. There kind of already is a "generic email framework" provided via the Mailer system, Swiftmail library, XF email templates and bounce handling (albeit IMAP/POP based). The unsubscribe system in XF2 is very powerful too - so list cleaning (via bounces and unsubscribes and the "active users" setting) is already implemented.

The Swiftmailer library is quite powerful, flexible and widely used - so it's relatively easy to implement custom sending solutions - the extension points built in to XF2.0 make that quite straight forward.

Beyond that - I'm not sure what else could be built in which isn't service-specific. Most ESPs have quite different implementations which makes trying to build a generic service not really feasible - I think we already largely have that.

I think any further work the XF devs did in this area would have to be service-specific, which they may be reluctant to do.

 

[Mouth]

Should be find for smaller forums - but completely unsuitable for larger sites.

Exactly. Google Mail via API and Auth is a very quick and easy way for a forum with less than 2k emails per day (and I'd wager thats the majority) to be sending emails from a respected source without paying a subscription. Larger sites can use the other suggested services.

 

[Mouth]

Most ESPs have quite different implementations which makes trying to build a generic service not really feasible

All of them have official/supported PHP libraries, making framework integration quick and easy ...
https://github.com/ElasticEmail/ElasticEmail.WebApiClient-php
https://github.com/sendgrid/sendgrid-php
https://github.com/sendinblue/APIv3-php-library
https://documentation.mailgun.com/en/latest/libraries.html#php
https://docs.aws.amazon.com/aws-sdk-php/v2/guide/service-ses.html
https://github.com/SparkPost/php-sparkpost
https://bitbucket.org/mailchimp/mandrill-api-php
https://github.com/mailjet/mailjet-apiv3-php

And my personal favourite ESP subscription service ..
https://postmarkapp.com/loves/php

 

[Sim]

All of them have official/supported PHP libraries, making basic framework integration quick and easy ...

... fixed that for you.

For sure, sending emails is easy. But that's less than half the story. Dealing with bounced messages and other feedback loops, handling unsubscribe requests, differentiating between transactional and non-transactional emails, optimising for deliverability, managing dedicated IP addresses and facilitiating IP address warmup protocols - that all takes time.

I have spent a long time developing my SparkPost solution. Getting it to send emails was the easy part - dealing with the nuances and complexities of optimising the system was what took the longest.

Sending emails really is easy - ensuring those emails appear in people's inboxes and not their spam folders is a LOT more complicated, especially when you are sending a lot of emails and start to include non-transactional emails in the mix.

 

[Mouth]

... fixed that for you.

No, just interpreted it the way you wish.

Sending emails really is easy - ensuring those emails appear in people's inboxes and not their spam folders is a LOT more complicated, especially when you are sending a lot of emails and start to include non-transactional emails in the mix.

Which goes to the point of @nrep and @Alfa1 in suggesting XF do more than just SMTP sending and POP/IMAP bounce checking, supporting at least a couple of ESP's through WebAPI etc. so that we get better support and efficiency.

 

[nrep]

I think any further work the XF devs did in this area would have to be service-specific, which they may be reluctant to do.

Yep, it likely would need to include a few service specific providers (the ones @Alfa1 posted in #6 cover almost all of the likely services), but that's all we would need to be able to avoid the SMTP/IMAP method, which is a lot less reliable with regards to bounce handling and statistics compared to using the service APIs.

 

[DragonByte Tech]

The problem with Google mail is the limitations they place on sending - Gmail isn't intended for high volume mail sending.

The Gmail API enforces the daily mail sending limits described here: https://support.google.com/a/answer/166852

Basically, 2,000 messages per day is a hard limit which cannot be changed or worked around. Should be find for smaller forums - but completely unsuitable for larger sites.

Even a small forum that want to send a newsletter about a forum event or something wouldn't be able to use Gmail for this. Even if they only had 1800 members (a tiny tiny site by any standard), the admin wouldn't be able to guarantee there wouldn't be 200 transactional emails needing sent in the next 24 hours.

IIRC from my research I did find enterprise functionality from Gmail that would increase the sending limits, but they are prohibitively expensive compared to something like Amazon SES where 10k emails will cost you approx $0.02.


Fillip

 

[S Thomas]

No, just interpreted it the way you wish.


Which goes to the point of @nrep and @Alfa1 in suggesting XF do more than just SMTP sending and POP/IMAP bounce checking, supporting at least a couple of ESP's through WebAPI etc. so that we get better support and efficiency.

I can just repeat what I have said - with swiftmailer as the underlying email handler you already have the generic email framework sort of . The only thing the devs could do is to split the implementation of the xF side of swiftmailer into an interface or something like that so it would be pretty straight forward to implement another esp lib into xF. In other words :there is already this example of "integrate an esp" within xF. This swiftmailer Integration already has templates, bounce handlers and list handlers. This is a complete example like PayPal is. There is a generic framework and the pp integration. There is the kinda generic email handler (swiftmailer) and it's xF integration (which, aswell, is swiftmailer).
Yes, this means, as I said before, you would need to check the external libs if or how they handle bounced emails etc. But that is, as @Sim said, provider specific. So not much more to do here. It would have literally nothing to do with xF - the xF part already exists.

 

[S Thomas]

All of them have official/supported PHP libraries, making framework integration quick and easy ...
https://github.com/ElasticEmail/ElasticEmail.WebApiClient-php
https://github.com/sendgrid/sendgrid-php
https://github.com/sendinblue/APIv3-php-library
https://documentation.mailgun.com/en/latest/libraries.html#php
https://docs.aws.amazon.com/aws-sdk-php/v2/guide/service-ses.html
https://github.com/SparkPost/php-sparkpost
https://bitbucket.org/mailchimp/mandrill-api-php
https://github.com/mailjet/mailjet-apiv3-php

And my personal favourite ESP subscription service ..
https://postmarkapp.com/loves/php

Yep, it likely would need to include a few service specific providers (the ones @Alfa1 posted in #6 cover almost all of the likely services), but that's all we would need to be able to avoid the SMTP/IMAP method, which is a lot less reliable with regards to bounce handling and statistics compared to using the service APIs.

If I'm not mistaken, Web Apis always have hard(er) limits over the classic email. Integrating an esp over an email specific protocol should have its advantages, but yea, probably disadvantages aswell.