Most forums need to deal with members whose email address becomes invalid. Sometimes the user will go ahead and change it, but other times your forum will fire off emails to this now invalid address. When this happens, if you have a bounce email address setup in XenForo (and possibly with the -f option enabled in the mail configuration), you will receive an email telling you about the failure.
Email delivery failures can happen for a wide variety of reasons. If you continually send to an address that has been disabled, for example, this can count against your domain/IP's email reputation. A lower email reputation can lead to reduced email deliverability as your domain may appear to be a spammer.
Therefore, when you are notified that an email can't be delivered, you may want to prevent further emails from being sent to that user. XenForo 1.3 added a new user state that represents an invalid email, but you had to manually change users into this state. With a decent size forum, this could be tedious and time consuming at best and impossible to manage at worst.
To handle this, XenForo 1.4 implements an automated bounce email processing system. This is a fairly detailed and technical system, but it allows an important maintenance action to be automated.
Automated Bounce Parsing
The core component of this system is a process that reads emails from a specific account and then parses those emails to determine if they are bounce messages.
As bounce notifications will be sent to the value you enter for the "Bounced Email Address", you would enter the POP3 or IMAP details for that account here. Once you do that, the system will automatically start checking this account for bounce messages. Note that the emails in this account will be removed after reading so this account needs to be dedicated to bounces from your forum only.
Every email that is read from this account will be analyzed to determine if it fits a known type and the appropriate action will be taken (changing the user state to indicate that the email is invalid, if appropriate). This is all displayed in the email bounce log in the control panel:
The action taken will depend on the content of the email. The sections below discuss why an email could be untrusted and the distinction between soft and hard bounces.
Bounce email notification parsing is notoriously difficult due to a wide range of servers that don't actually give good bounce responses. We have covered a number of common cases but this will certainly need to be expanded over time, so if you use this feature, keep an eye on the bounce log and let us know if you see emails that aren't being handled correctly. The "view original" link will take you to a raw output of the original email (headers and all encoded data). This data can be used to help us determine any additional cases to handle.
Note that you may get non-bounce emails to this address, such as delay notifications, challenge verifications and auto-replies. These will be logged, but they don't represent any errors so no action will be taken.
Bounce email logs will be maintained for 30 days.
VERP and Validation
While you would think that a bounce notification would always tell you who the email was sent to, you'd be wrong. Sometimes, the notification will only show you the address of the final recipient, and maybe they forwarded their email to another address. In rare cases, it might not list any address.
A common way around this is with a technique called Variable Envelope Return Path (VERP). The concept behind VERP is to encode the recipient email into the Return-Path header (which is where the bounce notification is sent).
Another important component is validating that the original email was actually sent by your forum. Without this, someone could manually create a bounce notification, send it to your bounce address, and trigger the bounce behavior for any email. By adding a validation hash into the return path, we can simply ignore any email that doesn't validate. This is what the "untrusted" component is in the email bounce log screenshot.
Let's look at a quick VERP example:
Keep in mind that the Return-Path is never shown to the end user (unless they look at the raw email contents). The Return-Path and the From/Sender values are always unrelated.
We would recommend enabling VERP support if you enable the automated bounce handling option. However, it does require that your SMTP server ignores anything after the + sign. This is quite common these days; Gmail does it, for example.
If you don't (or can't) enable the VERP option, we will embed this data in the original email via another header. Some bounce notifications include the original message (or the headers from it) and we can parse this data out of that. However, it may not be as reliable as the VERP option.
Soft and Hard Bounces
Bounce notifications report a wide range of issues. Some of these are unlikely to ever be resolved and others may be temporary or specific to a single message. This is where a soft and hard bounce distinction becomes necessary.
When a hard bounce happens, action will be taken immediately (setting the user state to indicate that their email is invalid). These issues are unlikely to ever be resolved, except maybe through dumb luck. Most commonly, this will be an invalid mailbox (the part to the left of the @ in the address).
Soft bounces include:
The criteria is designed to give the recipient (or their server admin) a chance to resolve the issue. If the limit was based solely on the number of emails, a flurry of messages in a short time span could trigger the soft bounce threshold.
Using the values in the screenshot, the email would only be considered invalid if:
Phew, that was a long one. Let us know if you have any questions or thoughts and keep an eye out for more 1.4 features being revealed soon.
Just a reminder: Please do not post suggestions in this thread (even if you feel they are related). Use the dedicated suggestion forum so they can be tracked; suggestions made in this thread are unlikely to be implemented.
Email delivery failures can happen for a wide variety of reasons. If you continually send to an address that has been disabled, for example, this can count against your domain/IP's email reputation. A lower email reputation can lead to reduced email deliverability as your domain may appear to be a spammer.
Therefore, when you are notified that an email can't be delivered, you may want to prevent further emails from being sent to that user. XenForo 1.3 added a new user state that represents an invalid email, but you had to manually change users into this state. With a decent size forum, this could be tedious and time consuming at best and impossible to manage at worst.
To handle this, XenForo 1.4 implements an automated bounce email processing system. This is a fairly detailed and technical system, but it allows an important maintenance action to be automated.
Automated Bounce Parsing
The core component of this system is a process that reads emails from a specific account and then parses those emails to determine if they are bounce messages.
As bounce notifications will be sent to the value you enter for the "Bounced Email Address", you would enter the POP3 or IMAP details for that account here. Once you do that, the system will automatically start checking this account for bounce messages. Note that the emails in this account will be removed after reading so this account needs to be dedicated to bounces from your forum only.
Every email that is read from this account will be analyzed to determine if it fits a known type and the appropriate action will be taken (changing the user state to indicate that the email is invalid, if appropriate). This is all displayed in the email bounce log in the control panel:
The action taken will depend on the content of the email. The sections below discuss why an email could be untrusted and the distinction between soft and hard bounces.
Bounce email notification parsing is notoriously difficult due to a wide range of servers that don't actually give good bounce responses. We have covered a number of common cases but this will certainly need to be expanded over time, so if you use this feature, keep an eye on the bounce log and let us know if you see emails that aren't being handled correctly. The "view original" link will take you to a raw output of the original email (headers and all encoded data). This data can be used to help us determine any additional cases to handle.
Note that you may get non-bounce emails to this address, such as delay notifications, challenge verifications and auto-replies. These will be logged, but they don't represent any errors so no action will be taken.
Bounce email logs will be maintained for 30 days.
VERP and Validation
While you would think that a bounce notification would always tell you who the email was sent to, you'd be wrong. Sometimes, the notification will only show you the address of the final recipient, and maybe they forwarded their email to another address. In rare cases, it might not list any address.
A common way around this is with a technique called Variable Envelope Return Path (VERP). The concept behind VERP is to encode the recipient email into the Return-Path header (which is where the bounce notification is sent).
Another important component is validating that the original email was actually sent by your forum. Without this, someone could manually create a bounce notification, send it to your bounce address, and trigger the bounce behavior for any email. By adding a validation hash into the return path, we can simply ignore any email that doesn't validate. This is what the "untrusted" component is in the email bounce log screenshot.
Let's look at a quick VERP example:
- Bounce handler address: bounce@yourdomain.com
- Email recipient: user@example.com
Keep in mind that the Return-Path is never shown to the end user (unless they look at the raw email contents). The Return-Path and the From/Sender values are always unrelated.
We would recommend enabling VERP support if you enable the automated bounce handling option. However, it does require that your SMTP server ignores anything after the + sign. This is quite common these days; Gmail does it, for example.
If you don't (or can't) enable the VERP option, we will embed this data in the original email via another header. Some bounce notifications include the original message (or the headers from it) and we can parse this data out of that. However, it may not be as reliable as the VERP option.
Soft and Hard Bounces
Bounce notifications report a wide range of issues. Some of these are unlikely to ever be resolved and others may be temporary or specific to a single message. This is where a soft and hard bounce distinction becomes necessary.
When a hard bounce happens, action will be taken immediately (setting the user state to indicate that their email is invalid). These issues are unlikely to ever be resolved, except maybe through dumb luck. Most commonly, this will be an invalid mailbox (the part to the left of the @ in the address).
Soft bounces include:
- The recipient's mailbox being full
- DNS/connectivity issues
- The message being too long (as other messages may go through)
The criteria is designed to give the recipient (or their server admin) a chance to resolve the issue. If the limit was based solely on the number of emails, a flurry of messages in a short time span could trigger the soft bounce threshold.
Using the values in the screenshot, the email would only be considered invalid if:
- We received 3 or more bounce notifications
- The bounce notifications happened on 3 or more distinct days
- The amount of time between the first and last bounce is at least 5 days
Phew, that was a long one. Let us know if you have any questions or thoughts and keep an eye out for more 1.4 features being revealed soon.
Just a reminder: Please do not post suggestions in this thread (even if you feel they are related). Use the dedicated suggestion forum so they can be tracked; suggestions made in this thread are unlikely to be implemented.
