XF 1.4 Automated Bounce Email Handling

Most forums need to deal with members whose email address becomes invalid. Sometimes the user will go ahead and change it, but other times your forum will fire off emails to this now invalid address. When this happens, if you have a bounce email address setup in XenForo (and possibly with the -f option enabled in the mail configuration), you will receive an email telling you about the failure.

Email delivery failures can happen for a wide variety of reasons. If you continually send to an address that has been disabled, for example, this can count against your domain/IP's email reputation. A lower email reputation can lead to reduced email deliverability as your domain may appear to be a spammer.

Therefore, when you are notified that an email can't be delivered, you may want to prevent further emails from being sent to that user. XenForo 1.3 added a new user state that represents an invalid email, but you had to manually change users into this state. With a decent size forum, this could be tedious and time consuming at best and impossible to manage at worst.

To handle this, XenForo 1.4 implements an automated bounce email processing system. This is a fairly detailed and technical system, but it allows an important maintenance action to be automated.

Automated Bounce Parsing
ss-2014-07-11_17-27-16.webp

The core component of this system is a process that reads emails from a specific account and then parses those emails to determine if they are bounce messages.

As bounce notifications will be sent to the value you enter for the "Bounced Email Address", you would enter the POP3 or IMAP details for that account here. Once you do that, the system will automatically start checking this account for bounce messages. Note that the emails in this account will be removed after reading so this account needs to be dedicated to bounces from your forum only.

Every email that is read from this account will be analyzed to determine if it fits a known type and the appropriate action will be taken (changing the user state to indicate that the email is invalid, if appropriate). This is all displayed in the email bounce log in the control panel:

ss-2014-07-11_17-38-46.webp


The action taken will depend on the content of the email. The sections below discuss why an email could be untrusted and the distinction between soft and hard bounces.

Bounce email notification parsing is notoriously difficult due to a wide range of servers that don't actually give good bounce responses. We have covered a number of common cases but this will certainly need to be expanded over time, so if you use this feature, keep an eye on the bounce log and let us know if you see emails that aren't being handled correctly. The "view original" link will take you to a raw output of the original email (headers and all encoded data). This data can be used to help us determine any additional cases to handle.

Note that you may get non-bounce emails to this address, such as delay notifications, challenge verifications and auto-replies. These will be logged, but they don't represent any errors so no action will be taken.

Bounce email logs will be maintained for 30 days.

VERP and Validation
While you would think that a bounce notification would always tell you who the email was sent to, you'd be wrong. Sometimes, the notification will only show you the address of the final recipient, and maybe they forwarded their email to another address. In rare cases, it might not list any address.

A common way around this is with a technique called Variable Envelope Return Path (VERP). The concept behind VERP is to encode the recipient email into the Return-Path header (which is where the bounce notification is sent).

Another important component is validating that the original email was actually sent by your forum. Without this, someone could manually create a bounce notification, send it to your bounce address, and trigger the bounce behavior for any email. By adding a validation hash into the return path, we can simply ignore any email that doesn't validate. This is what the "untrusted" component is in the email bounce log screenshot.

Let's look at a quick VERP example:
  • Bounce handler address: bounce@yourdomain.com
  • Email recipient: user@example.com
The VERP may look like: bounce+1234abcd+user=example.com@yourdomain.com. The "1234abcd" part is the validation key that is generated using a secret key.

Keep in mind that the Return-Path is never shown to the end user (unless they look at the raw email contents). The Return-Path and the From/Sender values are always unrelated.

ss-2014-07-11_17-58-14.webp


We would recommend enabling VERP support if you enable the automated bounce handling option. However, it does require that your SMTP server ignores anything after the + sign. This is quite common these days; Gmail does it, for example.

If you don't (or can't) enable the VERP option, we will embed this data in the original email via another header. Some bounce notifications include the original message (or the headers from it) and we can parse this data out of that. However, it may not be as reliable as the VERP option.

Soft and Hard Bounces
Bounce notifications report a wide range of issues. Some of these are unlikely to ever be resolved and others may be temporary or specific to a single message. This is where a soft and hard bounce distinction becomes necessary.

When a hard bounce happens, action will be taken immediately (setting the user state to indicate that their email is invalid). These issues are unlikely to ever be resolved, except maybe through dumb luck. Most commonly, this will be an invalid mailbox (the part to the left of the @ in the address).

Soft bounces include:
  • The recipient's mailbox being full
  • DNS/connectivity issues
  • The message being too long (as other messages may go through)
Soft bounces don't trigger the user state change until multiple messages bounce. The exact behavior is defined by a set of criteria:
ss-2014-07-11_18-06-50.webp

The criteria is designed to give the recipient (or their server admin) a chance to resolve the issue. If the limit was based solely on the number of emails, a flurry of messages in a short time span could trigger the soft bounce threshold.

Using the values in the screenshot, the email would only be considered invalid if:
  • We received 3 or more bounce notifications
  • The bounce notifications happened on 3 or more distinct days
  • The amount of time between the first and last bounce is at least 5 days
These values can be tuned to fit your needs and how strict you want to be.




Phew, that was a long one. Let us know if you have any questions or thoughts and keep an eye out for more 1.4 features being revealed soon.

Just a reminder: Please do not post suggestions in this thread (even if you feel they are related). Use the dedicated suggestion forum so they can be tracked; suggestions made in this thread are unlikely to be implemented.
 
I've been very vocal about the importance of handling bounced email management since becoming a XenForo customer; it's something many administrators don't think about, but it can cause some major problems as a community grows.

It will be so cool to have this functionality built-in and will save a lot of people a lot of headaches over the coming years. Very nice implementation, Mike! :D
 
Will there be any way to import bounced users into xf or to mass move users to bounced state within xf?

For example: I currently have 8500 users with bounced email (I automatically delete all 0 post bounced users). Is or will there be a way to apply bounced state to these users?
 
As any way you've done it would be proprietary to you, you'll need to do something custom to fit your needs. You just need to set the user_state for these users accordingly (provided the state is currently valid).
 
I asked on a separate thread but didn't get a reply

Is there anyway that XF can be configured so that it will accept emails directly from a php script rather than having XF check an inbox?

In this scenario, a php script is called with the bounced email as the raw input, XF would then action in exactly same as way as if it had just read the email in an inbox. It is very easy to set up postfix to deliver an email to a php script rather than a mailbox.

I've really got very serious concerns about the current proposed method running on my large forum.
 
There is no built in option to do that. I doubt it would be hard to write something for your needs to do it, but I don't see the fundamental issues with checking a mailbox.
 
I don't see the fundamental issues with checking a mailbox.
For starters there is the mailbox - gmail, for example like many providers, do not permit mailboxes that are used by mechanical means. While they aren't particularly strict about it, they will throw the odd curve ball, such as making you go through some manual process to show you are a human if they suspect anything - I've been there, I've done that - mail stops for some unapparent reason. Having also done email hosting in the past I know how difficult running a well run mailbox is - my advise is - take every opportunity to avoid it
Secondly there is the issue of scaling. We will do email campaigns where we send tens of thousands of emails in a short period of time. We run multiple servers working in unison on Xenforo. Any email email campaign is likely to result in a sudden influx of thousands of bounced emails (sad but true in our case). As we run XF over auto-scaled servers at AWS having one server suddenly being pre-occupied with a major task will play havoc on our systems as the expectation is that the work load will be even between servers. We have a utility server for this sort of stuff, and that is the server we would be wanting to use for this purpose and we'd like to be able to use it.
Thirdly you are introducing a dependency on a third party. If whatever mail provider I choose is having issues, then potentially our XF system can suffer delays due to issues at the mail provider, never mind the network overhead incurred in getting emails.
Fourthly, security, your introducing a process where data gets ingested by XF by potentially untrusted third parties.

Whilst I run one of XF's biggest forums, I've been around long enough to know what I am requesting is something that I would strongly recommend for any forum owner regardless of the size of their forum if they control their own server or VPS. It makes XF more reliable and more secure and is simple for any forum owner to configure and implement.
 
Any email email campaign is likely to result in a sudden influx of thousands of bounced emails (sad but true in our case).

I'd just like to add that it's time we stop apologizing for having a bunch of email addresses that bounce. In other words, it's not just your case. It's a fact of life for anyone who runs an internet business that collects email addresses. People's email addresses change, sometimes with regularity.
 
I need something cleared up. Will the user get any message or notice saying their email is broken or does this system just quietly not send email to that user anymore?
 
The member's account state will be set to "Email invalid (bounced)" and a message will be displayed advising them to update their email address.
 
What are the phrases that the users see when they get their email marked as bounced? I mean there must be a phrase with something like "Your email is invalid, please fix." I need to know what phrases are shown to the user so I can translate.
 
Top Bottom